Pearls of Luthra
Pearls of Luthra is the first book by Brian Jacques and I think I am going to be a fan of his work. This particular book you have to be wary of. While it is a beautiful book with quite a few illustrations, I have to warn that if you are somebody who feels hungry at the very mention of food, then you will be hungry throughout the book. There isn t a single page where food isn t mentioned and not just any kind of food, the kind of food that is geared towards sweet tooth. So if you fancy tarts or chocolates or anything sweet you will right at home. The book also touches upon various teas and wines and various liquors but food is where it shines in literally. The tale is very much like a Harry Potter adventure but isn t as dark as HP was. In fact, apart from one death and one ear missing rest of our heroes and heroines and there are quite a few. I don t want to give too much away as it s a book to be treasured.
Dahaad
Dahaad (the roar) is Sonakshi Sinha s entry in OTT/Web Series. The stage is set somewhere in North India while the exploits are based on a real life person called Cyanide Mohan who killed 20 women between 2005-2009. In the web series however, the antagonist s crimes are done over a period of 12 years and has 29 women as his victims. Apart from that it s pretty much a copy of what was done by the person above. It s a melting pot of a series which quite a few stories enmeshed along with the main one. The main onus and plot of the movie is about women from lower economic and caste order whose families want them to be wed but cannot due to huge demands for dowry. Now in such a situation, if a person were to give them a bit of attention, promise marriage and ask them to steal a bit and come with him and whatever, they will do it. The same modus operandi was done by Cynaide Mohan. He had a car that was not actually is but used it show off that he s from a richer background, entice the women, have sex, promise marriage and in the morning after pill there will be cynaide which the women unwittingly will consume.
This is also framed by the protagonist Sonakshi Sinha to her mother as her mother is also forcing her to get married as she is becoming older. She shows some of the photographs of the victims and says that while the perpetrator is guilty but so is the overall society that puts women in such vulnerable positions. AFAIK, that is still the state of things. In fact, there is a series called Indian Matchmaking that has all the snobbishness that you want. How many people could have a lifestyle like the ones shown in that, less than 2% of the population. It s actually shows like the above that make the whole thing even more precarious
Apart from it, the show also shows prejudice about caste and background. I wouldn t go much into it as it s worth seeing and experiencing.
Tetris
Tetris in many a ways is a story of greed. It s also a story of a lone inventor who had to wait almost 20 odd years to profit from his invention. Forbes does a marvelous job of giving some more background and foreground info. about Tetris, the inventor and the producer that went to strike it rich. It also does share about copyright misrepresentation happens but does nothing to address it. Could talk a whole lot but better to see the movie and draw your own conclusions. For me it was 4/5.
Discord
Discord became Discord 2.0 and is a blank to me. A blank page. Can t do anything. First I thought it was a bug. Waited for a few days as sometimes webservices do fix themselves. But two weeks on and it still wasn t fixed then decided to look under. One of the tools in Firefox is Web Developer Tools ( CTRL+Shift+I) that tells you if an element of a page is not appearing or at least gives you a hint. To me it gave me the following
Content Security Policy: Ignoring 'unsafe-inline' within script-src or style-src: nonce-source or hash-source specified Content Security Policy: The page s settings blocked the loading of a resource at data:text/css,%0A%20%20%20%20%20%20%20%2 ( style-src ). data:44:30 Content Security Policy: Ignoring 'unsafe-inline' within script-src or style-src: nonce-source or hash-source specified TypeError: AudioContext is not a constructor 138875 https://discord.com/assets/cbf3a75da6e6b6a4202e.js:262 l https://discord.com/assets/f5f0b113e28d4d12ba16.js:1ed46a18578285e5c048b.js:241:118
What is being done is dom.webaudio.enabled being disabled in Firefox.
Then on a hunch, searched on reddit and saw the following. Be careful while visiting the link as it s labelled NSFW although to my mind there wasn t anything remotely NSFW about it. They do mention using another tool AudioContext Fingerprint Defender which supposedly fakes or spoofs an id. As this add-on isn t tracked by Firefox privacy team it s hard for me to say anything positive or negative.
So, in the end I stopped using discord as the alternative was being tracked by them
Last but not the least, saw this about a week back. Sooner or later this had to happen as Elon tries to make money off Twitter.
Docker and other container systems by default restrict access to devices on the host. They used to do this with cgroups with the cgroup v1 system, however, the second version of cgroups removed this controller and the man page says:
Cgroup v2 device controller has no interface files and is implemented on top of cgroup BPF.
That is just awesome, nothing to see here, go look at the BPF documents if you have cgroup v2.
With cgroup v1 if you wanted to know what devices were permitted, you just would cat /sys/fs/cgroup/XX/devices.allow and you were done!
The kernel documentation is not very helpful, sure its something in BPF and has something to do with the cgroup BPF specifically, but what does that mean?
There doesn t seem to be an easy corresponding method to get the same information. So to see what restrictions a docker container has, we will have to:
Find what cgroup the programs running in the container belong to
Find what is the eBPF program ID that is attached to our container cgroup
Dump the eBPF program to a text file
Try to interpret the eBPF syntax
The last step is by far the most difficult.
Finding a container s cgroup
All containers have a short ID and a long ID. When you run the docker ps command, you get the short id. To get the long id you can either use the --no-trunc flag or just guess from the short ID. I usually do the second.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a3c53d8aaec2 debian:minicom "/bin/bash" 19 minutes ago Up 19 minutes inspiring_shannon
So the short ID is a3c53d8aaec2 and the long ID is a big ugly hex string starting with that. I generally just paste the relevant part in the next step and hit tab. For this container the cgroup is /sys/fs/cgroup/system.slice/docker-a3c53d8aaec23c256124f03d208732484714219c8b5f90dc1c3b4ab00f0b7779.scope/ Notice that the last directory has docker- then the short ID.
If you re not sure of the exact path. The /sys/fs/cgroup is the cgroup v2 mount point which can be found with mount -t cgroup2 and then rest is the actual cgroup name. If you know the process running in the container then the cgroup column in ps will show you.
eBPF programs and cgroups
Next we will need to get the eBPF program ID that is attached to our recently found cgroup. To do this, we will need to use the bpftool. One thing that threw me for a long time is when the tool talks about a program or a PROG ID they are talking about the eBPF programs, not your processes! With that out of the way, let s find the prog id.
$ sudo bpftool cgroup list /sys/fs/cgroup/system.slice/docker-a3c53d8aaec23c256124f03d208732484714219c8b5f90dc1c3b4ab00f0b7779.scope/
ID AttachType AttachFlags Name
90 cgroup_device multi
Our cgroup is attached to eBPF prog with ID of 90 and the type of program is cgroup _device.
Dumping the eBPF program
Next, we need to get the actual code that is run every time a process running in the cgroup tries to access a device. The program will take some parameters and will return either a 1 for yes you are allowed or a zero for permission denied. Don t use the file option as it dumps the program in binary format. The text version is hard enough to understand.
sudo bpftool prog dump xlated id 90 > myebpf.txt
Congratulations! You now have the eBPF program in a human-readable (?) format.
Interpreting the eBPF program
The eBPF format as dumped is not exactly user friendly. It probably helps to first go and look at an example program to see what is going on. You ll see that the program splits type (lower 4 bytes) and access (higher 4 bytes) and then does comparisons on those values. The eBPF has something similar:
What we find is that once we get past the first few lines filtering the given value that the comparison lines have:
r2 is the device type, 1 is block, 2 is character.
r3 is the device access, it s used with r1 for comparisons after masking the relevant bits. mknod, read and write are 1,2 and 3 respectively.
r4 is the major number
r5 is the minor number
For a even pretty simple setup, you are going to have around 60 lines of eBPF code to look at. Luckily, you ll often find the lines for the command options you added will be near the end, which makes it easier. For example:
This is a container using the option --device-cgroup-rule='c 100:42 rwm'. It is checking if r2 (device type) is 2 (char) and r4 (major device number) is 0x64 or 100 and r5 (minor device number) is 0x2a or 42. If any of those are not true, move to the next section, otherwise return with 1 (permit). We have all access modes permitted so it doesn t check for it.
The previous example has all permissions for our device with id 100:42, what about if we only want write access with the option --device-cgroup-rule='c 100:42 r'. The resulting eBPF is:
The code is almost the same but we are checking that w3 only has the second bit set, which is for reading, effectively checking for X==X&2. It s a cautious approach meaning no access still passes but multiple bits set will fail.
The device option
docker run allows you to specify files you want to grant access to your containers with the --device flag. This flag actually does two things. The first is to great the device file in the containers /dev directory, effectively doing a mknod command. The second thing is to adjust the eBPF program. If the device file we specified actually did have a major number of 100 and a minor of 42, the eBPF would look exactly like the above snippets.
What about privileged?
So we have used the direct cgroup options here, what does the --privileged flag do? This lets the container have full access to all the devices (if the user running the process is allowed). Like the --device flag, it makes the device files as well, but what does the filtering look like? We still have a cgroup but the eBPF program is greatly simplified, here it is in full:
We are happy to share that the RcppSimdJson
package has been updated to release 0.1.10.
RcppSimdJson
wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire and collaborators. Via
very clever algorithmic engineering to obtain largely branch-free code,
coupled with modern C++ and newer compiler instructions, it results in
parsing gigabytes of JSON parsed per second which is quite
mindboggling. The best-case performance is faster than CPU speed as
use of parallel SIMD instructions and careful branch avoidance can lead
to less than one cpu cycle per byte parsed; see the video of the talk by Daniel Lemire
at QCon.
This release updates the underlying simdjson library to
version 3.1.8 (also made today). Otherwise we only made a minor edit to
the README and adjusted one tweek for code coverage.
The (very short) NEWS entry for this release follows.
Changes in version 0.1.10
(2023-05-14)
simdjson was upgraded to version 3.1.8
(Dirk in #85).
Hello folks!
We recently received a case letting us know that Dataproc 2.1.1 was unable to write to a BigQuery table with a column of type JSON. Although the BigQuery connector for Spark has had support for JSON columns since 0.28.0, the Dataproc images on the 2.1 line still cannot create tables with JSON columns or write to existing tables with JSON columns.
The customer has graciously granted permission to share the code we developed to allow this operation. So if you are interested in working with JSON column tables on Dataproc 2.1 please continue reading!
Use the following gcloud command to create your single-node dataproc cluster:
The following file is the Scala code used to write JSON structured data to a BigQuery table using Spark. The file following this one can be executed from your single-node Dataproc cluster.
Main.scala
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.types. Metadata, StringType, StructField, StructType
import org.apache.spark.sql. Row, SaveMode, SparkSession
import org.apache.spark.sql.avro
import org.apache.avro.specific
val env = "x"
val my_bucket = "cjac-docker-on-yarn"
val my_table = "dataset.testavro2"
val spark = env match
case "local" =>
SparkSession
.builder()
.config("temporaryGcsBucket", my_bucket)
.master("local")
.appName("isssue_115574")
.getOrCreate()
case _ =>
SparkSession
.builder()
.config("temporaryGcsBucket", my_bucket)
.appName("isssue_115574")
.getOrCreate()
// create DF with some data
val someData = Seq(
Row(""" "name":"name1", "age": 10 """, "id1"),
Row(""" "name":"name2", "age": 20 """, "id2")
)
val schema = StructType(
Seq(
StructField("user_age", StringType, true),
StructField("id", StringType, true)
)
)
val avroFileName = s"gs://$ my_bucket /issue_115574/someData.avro"
val someDF = spark.createDataFrame(spark.sparkContext.parallelize(someData), schema)
someDF.write.format("avro").mode("overwrite").save(avroFileName)
val avroDF = spark.read.format("avro").load(avroFileName)
// set metadata
val dfJSON = avroDF
.withColumn("user_age_no_metadata", col("user_age"))
.withMetadata("user_age", Metadata.fromJson(""" "sqlType":"JSON" """))
dfJSON.show()
dfJSON.printSchema
// write to BigQuery
dfJSON.write.format("bigquery")
.mode(SaveMode.Overwrite)
.option("writeMethod", "indirect")
.option("intermediateFormat", "avro")
.option("useAvroLogicalTypes", "true")
.option("table", my_table)
.save()
#!/bin/bash
PROJECT_ID=set-yours-here
DATASET_NAME=dataset
TABLE_NAME=testavro2
# We have to remove all of the existing spark bigquery jars from the local
# filesystem, as we will be using the symbols from the
# spark-3.3-bigquery-0.30.0.jar below. Having existing jar files on the
# local filesystem will result in those symbols having higher precedence
# than the one loaded with the spark-shell.
sudo find /usr -name 'spark*bigquery*jar' -delete
# Remove the table from the bigquery dataset if it exists
bq rm -f -t $PROJECT_ID:$DATASET_NAME.$TABLE_NAME
# Create the table with a JSON type column
bq mk --table $PROJECT_ID:$DATASET_NAME.$TABLE_NAME \
user_age:JSON,id:STRING,user_age_no_metadata:STRING
# Load the example Main.scala
spark-shell -i Main.scala \
--jars /usr/lib/spark/external/spark-avro.jar,gs://spark-lib/bigquery/spark-3.3-bigquery-0.30.0.jar
# Show the table schema when we use bq mk --table and then load the avro
bq query --use_legacy_sql=false \
"SELECT ddl FROM $DATASET_NAME.INFORMATION_SCHEMA.TABLES where table_name='$TABLE_NAME'"
# Remove the table so that we can see that the table is created should it not exist
bq rm -f -t $PROJECT_ID:$DATASET_NAME.$TABLE_NAME
# Dynamically generate a DataFrame, store it to avro, load that avro,
# and write the avro to BigQuery, creating the table if it does not already exist
spark-shell -i Main.scala \
--jars /usr/lib/spark/external/spark-avro.jar,gs://spark-lib/bigquery/spark-3.3-bigquery-0.30.0.jar
# Show that the table schema does not differ from one created with a bq mk --table
bq query --use_legacy_sql=false \
"SELECT ddl FROM $DATASET_NAME.INFORMATION_SCHEMA.TABLES where table_name='$TABLE_NAME'"
Google BigQuery has supported JSON data since October of 2022, but until now, it has not been possible, on generally available Dataproc clusters, to interact with these columns using the Spark BigQuery Connector.
JSON column type support was introduced in spark-bigquery-connector release 0.28.0.
Posted on April 30, 2023
A few months ago we may have bought a few ESP32-C3-DevKit-Lipo boards from Olimex.
Since every time I go back to working with them I ve forgotten how to do so, and my old notes on the fediverse are hard to find, this is the full procedure.
Install arduino from the distribution packages (version 1.8 is ok).
Under File Preferences, add the development URL to the Additional Boards Manager URLs field. (on 2023-04-30 that s https://espressif.github.io/arduino-esp32/package_esp32_dev_index.json).
Under Tools Board Boards Manager make sure that you install a version of esp32 by Espressif Systems that is above 2.0 (on 2023-04-30 there is a 2.0.8 that works).
Programming
Under Tools Board ESP32 Arduino select ESP32C3 Dev Module.
Under Tools USB CDC On Boot select Enabled.
You can now compile and upload your sketches.
If something goes wrong, to force the board to bootloader mode bring GPIO9 to GND.
Note that the serial port device /dev/ttyACM0 only appears when in bootloader mode, but uploading sketches and the serial monitor will still work even if the port is not set in the arduino IDE.
Results of the Debian Developer Survey, by Roberto C. S nchez
In 2022, Freexian polled Debian Developers
about the usage of money in Debian. More than 200 Debian Developers
graciously participated, providing useful and constructive answers.
Roberto and Utkarsh have worked on reviewing this feedback and
summarizing it in a report recently published and announced to the project.
DebConf 23 Website, by Stefano Rivera
In preparation for DebConf 23,
Stefano did some work on the DebConf website s registration system.
To support an expected large number of local registration requests,
and a limited venue size, Stefano added a review system for
registration requests.
There was also some infrastructure work for the website framework. We
use the same framework for miniconfs and DebConf, but without the full
registration system. Since last DebConf, we have migrated from a pure-JS
toolchain for the static assets, to django-compressor, to be friendlier
to contributors and have a simpler dependency setup. This required some
updates in the full-DebConf registration system that hadn t been noticed
yet in miniDebConfs. Finally, with Utkarsh, we started to wind up the
DebConf 22 travel bursary reimbursement process.
Debian Reimbursements Web App Progress, by Stefano Rivera
In a project funded by
Freexian s Project Funding initiative,
Stefano made some more progress on the
Debian Reimbursements Web App.
The first rough implementation core request lifecycle is almost
complete. Receipts can be collected and itemized, and the request
can be submitted for a reimbursement request.
Debian Printing, by Thorsten Alteholz
Due to the upcoming release, only bug fixing uploads are allowed
in this part of the release cycle and Thorsten did uploads of
three Debian Printing packages.
The upload of hplip was rather straightforward and five bugs
could be closed.
cups-filters suddenly started to FTBFS and thus got an RC bug. It
failed due to a compile error in a header file of some dependency.
Luckily the maintainer of that dependency knew that his package
now needed c++17, so the fix was to just remove an old compile
flag that forced the compiler to use c++0x. This flag was once
progressive but nowadays it is more of a hindrance than a help.
The third package upload was for cups, which got some translation
updates. Unfortunately this was the most tricky one as some
translations did not appear in the binary packages. After debugging
for some time, it turned out that the handling of links did not work
properly. Now the version in Bookworm will be the cups version with
the most translated man pages ever.
Miscellaneous contributions
Stefano Rivera updated a few Python modules in the Debian Python
Team, to the latest upstream versions.
Stefano Rivera reviewed the current patch series applied to Python
3.12, as an Arch package maintainer
had noticed that we dropped a patch by mistake, and reinstated it.
Anton Gladky prepared an upload of newer version (9.2.6) of vtk
library and uploaded it into the experimental due to a freeze.
VTK is the visualization kit - a library used mostly for scientific
and engineering applications to visualize complex objects. Transition
of dependent packages is planned on after-release phase.
Helmut Grohne, in the continual effort to improve Debian s cross-build
support, provided 22 cross-build patches to packages in the archive.
Today I finally found time to track down a useful radio frequency
scanner for my software defined radio. Just for fun I tried to locate
the radios used in the areas, and a good start would be to scan all
the frequencies to see what is in use. I've tried to find a useful
program earlier, but ran out of time before I managed to find a useful
tool. This time I was more successful, and after a few false leads I
found a description of
rtlsdr-scanner
over at the Kali site, and was able to track down
the
Kali package git repository to build a deb package for the
scanner. Sadly the package is missing from the Debian project itself,
at least in Debian Bullseye. Two runtime dependencies,
python-visvis
and
python-rtlsdr
had to be built and installed separately. Luckily 'gbp
buildpackage' handled them just fine and no further packages had
to be manually built. The end result worked out of the box after
installation.
My initial scans for FM channels worked just fine, so I knew the
scanner was functioning. But when I tried to scan every frequency
from 100 to 1000 MHz, the program stopped unexpectedly near the
completion. After some debugging I discovered USB software radio I
used rejected frequencies above 948 MHz, triggering a unreported
exception breaking the scan. Changing the scan to end at 957 worked
better. I similarly found the lower limit to be around 15, and ended
up with the following full scan:
Saving the scan did not work, but exporting it as a CSV file worked
just fine. I ended up with around 477k CVS lines with dB level for
the given frequency.
The save failure seem to be a missing UTF-8 encoding issue in the
python code. Will see if I can find time to send a patch
upstream
later to fix this exception:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/rtlsdr_scanner/main_window.py", line 485, in __on_save
save_plot(fullName, self.scanInfo, self.spectrum, self.locations)
File "/usr/lib/python3/dist-packages/rtlsdr_scanner/file.py", line 408, in save_plot
handle.write(json.dumps(data, indent=4))
TypeError: a bytes-like object is required, not 'str'
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/rtlsdr_scanner/main_window.py", line 485, in __on_save
save_plot(fullName, self.scanInfo, self.spectrum, self.locations)
File "/usr/lib/python3/dist-packages/rtlsdr_scanner/file.py", line 408, in save_plot
handle.write(json.dumps(data, indent=4))
TypeError: a bytes-like object is required, not 'str'
As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
I ve used hardware-backed OpenPGP keys since 2006 when I imported newly generated rsa1024 subkeys to a FSFE Fellowship card. This worked well for several years, and I recall buying more ZeitControl cards for multi-machine usage and backup purposes. As a side note, I recall being unsatisfied with the weak 1024-bit RSA subkeys at the time my primary key was a somewhat stronger 1280-bit RSA key created back in 2002 but OpenPGP cards at the time didn t support more than 1024 bit RSA, and were (and still often are) also limited to power-of-two RSA key sizes which I dislike.
I had my master key on disk with a strong password for a while, mostly to refresh expiration time of the subkeys and to sign other s OpenPGP keys. At some point I stopped carrying around encrypted copies of my master key. That was my main setup when I migrated to a new stronger RSA 3744 bit key with rsa2048 subkeys on a YubiKey NEO back in 2014. At that point, signing other s OpenPGP keys was a rare enough occurrence that I settled with bringing out my offline machine to perform this operation, transferring the public key to sign on USB sticks. In 2019 I re-evaluated my OpenPGP setup and ended up creating a offline Ed25519 key with subkeys on a FST-01G running Gnuk. My approach for signing other s OpenPGP keys were still to bring out my offline machine and sign things using the master secret using USB sticks for storage and transport. Which meant I almost never did that, because it took too much effort. So my 2019-era Ed25519 key still only has a handful of signatures on it, since I had essentially stopped signing other s keys which is the traditional way of getting signatures in return.
None of this caused any critical problem for me because I continued to use my old 2014-era RSA3744 key in parallel with my new 2019-era Ed25519 key, since too many systems didn t handle Ed25519. However, during 2022 this changed, and the only remaining environment that I still used my RSA3744 key for was in Debian and they require OpenPGP signatures on the new key to allow it to replace an older key. I was in denial about this sub-optimal solution during 2022 and endured its practical consequences, having to use the YubiKey NEO (which I had replaced with a permanently inserted YubiKey Nano at some point) for Debian-related purposes alone.
In December 2022 I bought a new laptop and setup a FST-01SZ with my Ed25519 key, and while I have taken a vacation from Debian, I continue to extend the expiration period on the old RSA3744-key in case I will ever have to use it again, so the overall OpenPGP setup was still sub-optimal. Having two valid OpenPGP keys at the same time causes people to use both for email encryption (leading me to have to use both devices), and the WKD Key Discovery protocol doesn t like two valid keys either. At FOSDEM 23 I ran into Andre Heinecke at GnuPG and I couldn t help complain about how complex and unsatisfying all OpenPGP-related matters were, and he mildly ignored my rant and asked why I didn t put the master key on another smartcard. The comment sunk in when I came home, and recently I connected all the dots and this post is a summary of what I did to move my offline OpenPGP master key to a Nitrokey Start.
First a word about device choice, I still prefer to use hardware devices that are as compatible with free software as possible, but the FST-01G or FST-01SZ are no longer easily available for purchase. I got a comment about Nitrokey start in my last post, and had two of them available to experiment with. There are things to dislike with the Nitrokey Start compared to the YubiKey (e.g., relative insecure chip architecture, the bulkier form factor and lack of FIDO/U2F/OATH support) but as far as I know there is no more widely available owner-controlled device that is manufactured for an intended purpose of implementing an OpenPGP card. Thus it hits the sweet spot for me.
Nitrokey Start
The first step is to run latest firmware on the Nitrokey Start for bug-fixes and important OpenSSH 9.0 compatibility and there are reproducible-built firmware published that you can install using pynitrokey. I run Trisquel 11 aramo on my laptop, which does not include the Python Pip package (likely because it promotes installing non-free software) so that was a slight complication. Building the firmware locally may have worked, and I would like to do that eventually to confirm the published firmware, however to save time I settled with installing the Ubuntu 22.04 packages on my machine:
$ sha256sum python3-pip*
ded6b3867a4a4cbaff0940cab366975d6aeecc76b9f2d2efa3deceb062668b1c python3-pip_22.0.2+dfsg-1ubuntu0.2_all.deb
e1561575130c41dc3309023a345de337e84b4b04c21c74db57f599e267114325 python3-pip-whl_22.0.2+dfsg-1ubuntu0.2_all.deb
$ doas dpkg -i python3-pip*
...
$ doas apt install -f
...
$
Installing pynitrokey downloaded a bunch of dependencies, and it would be nice to audit the license and security vulnerabilities for each of them. (Verbose output below slightly redacted.)
jas@kaka:~$ pip3 install --user pynitrokey
Collecting pynitrokey
Downloading pynitrokey-0.4.34-py3-none-any.whl (572 kB)
Collecting frozendict~=2.3.4
Downloading frozendict-2.3.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (113 kB)
Requirement already satisfied: click<9,>=8.0.0 in /usr/lib/python3/dist-packages (from pynitrokey) (8.0.3)
Collecting ecdsa
Downloading ecdsa-0.18.0-py2.py3-none-any.whl (142 kB)
Collecting python-dateutil~=2.7.0
Downloading python_dateutil-2.7.5-py2.py3-none-any.whl (225 kB)
Collecting fido2<2,>=1.1.0
Downloading fido2-1.1.0-py3-none-any.whl (201 kB)
Collecting tlv8
Downloading tlv8-0.10.0.tar.gz (16 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: certifi>=14.5.14 in /usr/lib/python3/dist-packages (from pynitrokey) (2020.6.20)
Requirement already satisfied: pyusb in /usr/lib/python3/dist-packages (from pynitrokey) (1.2.1.post1)
Collecting urllib3~=1.26.7
Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting spsdk<1.8.0,>=1.7.0
Downloading spsdk-1.7.1-py3-none-any.whl (684 kB)
Collecting typing_extensions~=4.3.0
Downloading typing_extensions-4.3.0-py3-none-any.whl (25 kB)
Requirement already satisfied: cryptography<37,>=3.4.4 in /usr/lib/python3/dist-packages (from pynitrokey) (3.4.8)
Collecting intelhex
Downloading intelhex-2.3.0-py2.py3-none-any.whl (50 kB)
Collecting nkdfu
Downloading nkdfu-0.2-py3-none-any.whl (16 kB)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from pynitrokey) (2.25.1)
Collecting tqdm
Downloading tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting nrfutil<7,>=6.1.4
Downloading nrfutil-6.1.7.tar.gz (845 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: cffi in /usr/lib/python3/dist-packages (from pynitrokey) (1.15.0)
Collecting crcmod
Downloading crcmod-1.7.tar.gz (89 kB)
Preparing metadata (setup.py) ... done
Collecting libusb1==1.9.3
Downloading libusb1-1.9.3-py3-none-any.whl (60 kB)
Collecting pc_ble_driver_py>=0.16.4
Downloading pc_ble_driver_py-0.17.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.9 MB)
Collecting piccata
Downloading piccata-2.0.3-py3-none-any.whl (21 kB)
Collecting protobuf<4.0.0,>=3.17.3
Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
Collecting pyserial
Downloading pyserial-3.5-py2.py3-none-any.whl (90 kB)
Collecting pyspinel>=1.0.0a3
Downloading pyspinel-1.0.3.tar.gz (58 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from nrfutil<7,>=6.1.4->pynitrokey) (5.4.1)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil~=2.7.0->pynitrokey) (1.16.0)
Collecting pylink-square<0.11.9,>=0.8.2
Downloading pylink_square-0.11.1-py2.py3-none-any.whl (78 kB)
Collecting jinja2<3.1,>=2.11
Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting bincopy<17.11,>=17.10.2
Downloading bincopy-17.10.3-py3-none-any.whl (17 kB)
Collecting fastjsonschema>=2.15.1
Downloading fastjsonschema-2.16.3-py3-none-any.whl (23 kB)
Collecting astunparse<2,>=1.6
Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting oscrypto~=1.2
Downloading oscrypto-1.3.0-py2.py3-none-any.whl (194 kB)
Collecting deepmerge==0.3.0
Downloading deepmerge-0.3.0-py2.py3-none-any.whl (7.6 kB)
Collecting pyocd<=0.31.0,>=0.28.3
Downloading pyocd-0.31.0-py3-none-any.whl (12.5 MB)
Collecting click-option-group<0.6,>=0.3.0
Downloading click_option_group-0.5.5-py3-none-any.whl (12 kB)
Collecting pycryptodome<4,>=3.9.3
Downloading pycryptodome-3.17-cp35-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
Collecting pyocd-pemicro<1.2.0,>=1.1.1
Downloading pyocd_pemicro-1.1.5-py3-none-any.whl (9.0 kB)
Requirement already satisfied: colorama<1,>=0.4.4 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (0.4.4)
Collecting commentjson<1,>=0.9
Downloading commentjson-0.9.0.tar.gz (8.7 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: asn1crypto<2,>=1.2 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (1.4.0)
Collecting pypemicro<0.2.0,>=0.1.9
Downloading pypemicro-0.1.11-py3-none-any.whl (5.7 MB)
Collecting libusbsio>=2.1.11
Downloading libusbsio-2.1.11-py3-none-any.whl (247 kB)
Collecting sly==0.4
Downloading sly-0.4.tar.gz (60 kB)
Preparing metadata (setup.py) ... done
Collecting ruamel.yaml<0.18.0,>=0.17
Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)
Collecting cmsis-pack-manager<0.3.0
Downloading cmsis_pack_manager-0.2.10-py2.py3-none-manylinux1_x86_64.whl (25.1 MB)
Collecting click-command-tree==1.1.0
Downloading click_command_tree-1.1.0-py3-none-any.whl (3.6 kB)
Requirement already satisfied: bitstring<3.2,>=3.1 in /usr/lib/python3/dist-packages (from spsdk<1.8.0,>=1.7.0->pynitrokey) (3.1.7)
Collecting hexdump~=3.3
Downloading hexdump-3.3.zip (12 kB)
Preparing metadata (setup.py) ... done
Collecting fire
Downloading fire-0.5.0.tar.gz (88 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/lib/python3/dist-packages (from astunparse<2,>=1.6->spsdk<1.8.0,>=1.7.0->pynitrokey) (0.37.1)
Collecting humanfriendly
Downloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Collecting argparse-addons>=0.4.0
Downloading argparse_addons-0.12.0-py3-none-any.whl (3.3 kB)
Collecting pyelftools
Downloading pyelftools-0.29-py2.py3-none-any.whl (174 kB)
Collecting milksnake>=0.1.2
Downloading milksnake-0.1.5-py2.py3-none-any.whl (9.6 kB)
Requirement already satisfied: appdirs>=1.4 in /usr/lib/python3/dist-packages (from cmsis-pack-manager<0.3.0->spsdk<1.8.0,>=1.7.0->pynitrokey) (1.4.4)
Collecting lark-parser<0.8.0,>=0.7.1
Downloading lark-parser-0.7.8.tar.gz (276 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: MarkupSafe>=2.0 in /usr/lib/python3/dist-packages (from jinja2<3.1,>=2.11->spsdk<1.8.0,>=1.7.0->pynitrokey) (2.0.1)
Collecting asn1crypto<2,>=1.2
Downloading asn1crypto-1.5.1-py2.py3-none-any.whl (105 kB)
Collecting wrapt
Downloading wrapt-1.15.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB)
Collecting future
Downloading future-0.18.3.tar.gz (840 kB)
Preparing metadata (setup.py) ... done
Collecting psutil>=5.2.2
Downloading psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
Collecting capstone<5.0,>=4.0
Downloading capstone-4.0.2-py2.py3-none-manylinux1_x86_64.whl (2.1 MB)
Collecting naturalsort<2.0,>=1.5
Downloading naturalsort-1.5.1.tar.gz (7.4 kB)
Preparing metadata (setup.py) ... done
Collecting prettytable<3.0,>=2.0
Downloading prettytable-2.5.0-py3-none-any.whl (24 kB)
Collecting intervaltree<4.0,>=3.0.2
Downloading intervaltree-3.1.0.tar.gz (32 kB)
Preparing metadata (setup.py) ... done
Collecting ruamel.yaml.clib>=0.2.6
Downloading ruamel.yaml.clib-0.2.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (485 kB)
Collecting termcolor
Downloading termcolor-2.2.0-py3-none-any.whl (6.6 kB)
Collecting sortedcontainers<3.0,>=2.0
Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: wcwidth in /usr/lib/python3/dist-packages (from prettytable<3.0,>=2.0->pyocd<=0.31.0,>=0.28.3->spsdk<1.8.0,>=1.7.0->pynitrokey) (0.2.5)
Building wheels for collected packages: nrfutil, crcmod, sly, tlv8, commentjson, hexdump, pyspinel, fire, intervaltree, lark-parser, naturalsort, future
Building wheel for nrfutil (setup.py) ... done
Created wheel for nrfutil: filename=nrfutil-6.1.7-py3-none-any.whl size=898520 sha256=de6f8803f51d6c26d24dc7df6292064a468ff3f389d73370433fde5582b84a10
Stored in directory: /home/jas/.cache/pip/wheels/39/2b/9b/98ab2dd716da746290e6728bdb557b14c1c9a54cb9ed86e13b
Building wheel for crcmod (setup.py) ... done
Created wheel for crcmod: filename=crcmod-1.7-cp310-cp310-linux_x86_64.whl size=31422 sha256=5149ac56fcbfa0606760eef5220fcedc66be560adf68cf38c604af3ad0e4a8b0
Stored in directory: /home/jas/.cache/pip/wheels/85/4c/07/72215c529bd59d67e3dac29711d7aba1b692f543c808ba9e86
Building wheel for sly (setup.py) ... done
Created wheel for sly: filename=sly-0.4-py3-none-any.whl size=27352 sha256=f614e413918de45c73d1e9a8dca61ca07dc760d9740553400efc234c891f7fde
Stored in directory: /home/jas/.cache/pip/wheels/a2/23/4a/6a84282a0d2c29f003012dc565b3126e427972e8b8157ea51f
Building wheel for tlv8 (setup.py) ... done
Created wheel for tlv8: filename=tlv8-0.10.0-py3-none-any.whl size=11266 sha256=3ec8b3c45977a3addbc66b7b99e1d81b146607c3a269502b9b5651900a0e2d08
Stored in directory: /home/jas/.cache/pip/wheels/e9/35/86/66a473cc2abb0c7f21ed39c30a3b2219b16bd2cdb4b33cfc2c
Building wheel for commentjson (setup.py) ... done
Created wheel for commentjson: filename=commentjson-0.9.0-py3-none-any.whl size=12092 sha256=28b6413132d6d7798a18cf8c76885dc69f676ea763ffcb08775a3c2c43444f4a
Stored in directory: /home/jas/.cache/pip/wheels/7d/90/23/6358a234ca5b4ec0866d447079b97fedf9883387d1d7d074e5
Building wheel for hexdump (setup.py) ... done
Created wheel for hexdump: filename=hexdump-3.3-py3-none-any.whl size=8913 sha256=79dfadd42edbc9acaeac1987464f2df4053784fff18b96408c1309b74fd09f50
Stored in directory: /home/jas/.cache/pip/wheels/26/28/f7/f47d7ecd9ae44c4457e72c8bb617ef18ab332ee2b2a1047e87
Building wheel for pyspinel (setup.py) ... done
Created wheel for pyspinel: filename=pyspinel-1.0.3-py3-none-any.whl size=65033 sha256=01dc27f81f28b4830a0cf2336dc737ef309a1287fcf33f57a8a4c5bed3b5f0a6
Stored in directory: /home/jas/.cache/pip/wheels/95/ec/4b/6e3e2ee18e7292d26a65659f75d07411a6e69158bb05507590
Building wheel for fire (setup.py) ... done
Created wheel for fire: filename=fire-0.5.0-py2.py3-none-any.whl size=116951 sha256=3d288585478c91a6914629eb739ea789828eb2d0267febc7c5390cb24ba153e8
Stored in directory: /home/jas/.cache/pip/wheels/90/d4/f7/9404e5db0116bd4d43e5666eaa3e70ab53723e1e3ea40c9a95
Building wheel for intervaltree (setup.py) ... done
Created wheel for intervaltree: filename=intervaltree-3.1.0-py2.py3-none-any.whl size=26119 sha256=5ff1def22ba883af25c90d90ef7c6518496fcd47dd2cbc53a57ec04cd60dc21d
Stored in directory: /home/jas/.cache/pip/wheels/fa/80/8c/43488a924a046b733b64de3fac99252674c892a4c3801c0a61
Building wheel for lark-parser (setup.py) ... done
Created wheel for lark-parser: filename=lark_parser-0.7.8-py2.py3-none-any.whl size=62527 sha256=3d2ec1d0f926fc2688d40777f7ef93c9986f874169132b1af590b6afc038f4be
Stored in directory: /home/jas/.cache/pip/wheels/29/30/94/33e8b58318aa05cb1842b365843036e0280af5983abb966b83
Building wheel for naturalsort (setup.py) ... done
Created wheel for naturalsort: filename=naturalsort-1.5.1-py3-none-any.whl size=7526 sha256=bdecac4a49f2416924548cae6c124c85d5333e9e61c563232678ed182969d453
Stored in directory: /home/jas/.cache/pip/wheels/a6/8e/c9/98cfa614fff2979b457fa2d9ad45ec85fa417e7e3e2e43be51
Building wheel for future (setup.py) ... done
Created wheel for future: filename=future-0.18.3-py3-none-any.whl size=492037 sha256=57a01e68feca2b5563f5f624141267f399082d2f05f55886f71b5d6e6cf2b02c
Stored in directory: /home/jas/.cache/pip/wheels/5e/a9/47/f118e66afd12240e4662752cc22cefae5d97275623aa8ef57d
Successfully built nrfutil crcmod sly tlv8 commentjson hexdump pyspinel fire intervaltree lark-parser naturalsort future
Installing collected packages: tlv8, sortedcontainers, sly, pyserial, pyelftools, piccata, naturalsort, libusb1, lark-parser, intelhex, hexdump, fastjsonschema, crcmod, asn1crypto, wrapt, urllib3, typing_extensions, tqdm, termcolor, ruamel.yaml.clib, python-dateutil, pyspinel, pypemicro, pycryptodome, psutil, protobuf, prettytable, oscrypto, milksnake, libusbsio, jinja2, intervaltree, humanfriendly, future, frozendict, fido2, ecdsa, deepmerge, commentjson, click-option-group, click-command-tree, capstone, astunparse, argparse-addons, ruamel.yaml, pyocd-pemicro, pylink-square, pc_ble_driver_py, fire, cmsis-pack-manager, bincopy, pyocd, nrfutil, nkdfu, spsdk, pynitrokey
WARNING: The script nitropy is installed in '/home/jas/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed argparse-addons-0.12.0 asn1crypto-1.5.1 astunparse-1.6.3 bincopy-17.10.3 capstone-4.0.2 click-command-tree-1.1.0 click-option-group-0.5.5 cmsis-pack-manager-0.2.10 commentjson-0.9.0 crcmod-1.7 deepmerge-0.3.0 ecdsa-0.18.0 fastjsonschema-2.16.3 fido2-1.1.0 fire-0.5.0 frozendict-2.3.5 future-0.18.3 hexdump-3.3 humanfriendly-10.0 intelhex-2.3.0 intervaltree-3.1.0 jinja2-3.0.3 lark-parser-0.7.8 libusb1-1.9.3 libusbsio-2.1.11 milksnake-0.1.5 naturalsort-1.5.1 nkdfu-0.2 nrfutil-6.1.7 oscrypto-1.3.0 pc_ble_driver_py-0.17.0 piccata-2.0.3 prettytable-2.5.0 protobuf-3.20.3 psutil-5.9.4 pycryptodome-3.17 pyelftools-0.29 pylink-square-0.11.1 pynitrokey-0.4.34 pyocd-0.31.0 pyocd-pemicro-1.1.5 pypemicro-0.1.11 pyserial-3.5 pyspinel-1.0.3 python-dateutil-2.7.5 ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.7 sly-0.4 sortedcontainers-2.4.0 spsdk-1.7.1 termcolor-2.2.0 tlv8-0.10.0 tqdm-4.65.0 typing_extensions-4.3.0 urllib3-1.26.15 wrapt-1.15.0
jas@kaka:~$
Then upgrading the device worked remarkable well, although I wish that the tool would have printed URLs and checksums for the firmware files to allow easy confirmation.
jas@kaka:~$ PATH=$PATH:/home/jas/.local/bin
jas@kaka:~$ nitropy start list
Command line tool to interact with Nitrokey devices 0.4.34
:: 'Nitrokey Start' keys:
FSIJ-1.2.15-5D271572: Nitrokey Nitrokey Start (RTM.12.1-RC2-modified)
jas@kaka:~$ nitropy start update
Command line tool to interact with Nitrokey devices 0.4.34
Nitrokey Start firmware update tool
Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
System: Linux, is_linux: True
Python: 3.10.6
Saving run log to: /tmp/nitropy.log.gc5753a8
Admin PIN:
Firmware data to be used:
- FirmwareType.REGNUAL: 4408, hash: ...b'72a30389' valid (from ...built/RTM.13/regnual.bin)
- FirmwareType.GNUK: 129024, hash: ...b'25a4289b' valid (from ...prebuilt/RTM.13/gnuk.bin)
Currently connected device strings:
Device:
Vendor: Nitrokey
Product: Nitrokey Start
Serial: FSIJ-1.2.15-5D271572
Revision: RTM.12.1-RC2-modified
Config: *:*:8e82
Sys: 3.0
Board: NITROKEY-START-G
initial device strings: [ 'name': '', 'Vendor': 'Nitrokey', 'Product': 'Nitrokey Start', 'Serial': 'FSIJ-1.2.15-5D271572', 'Revision': 'RTM.12.1-RC2-modified', 'Config': '*:*:8e82', 'Sys': '3.0', 'Board': 'NITROKEY-START-G' ]
Please note:
- Latest firmware available is:
RTM.13 (published: 2022-12-08T10:59:11Z)
- provided firmware: None
- all data will be removed from the device!
- do not interrupt update process - the device may not run properly!
- the process should not take more than 1 minute
Do you want to continue? [yes/no]: yes
...
Starting bootloader upload procedure
Device: Nitrokey Start FSIJ-1.2.15-5D271572
Connected to the device
Running update!
Do NOT remove the device from the USB slot, until further notice
Downloading flash upgrade program...
Executing flash upgrade...
Waiting for device to appear:
Wait 20 seconds.....
Downloading the program
Protecting device
Finish flashing
Resetting device
Update procedure finished. Device could be removed from USB slot.
Currently connected device strings (after upgrade):
Device:
Vendor: Nitrokey
Product: Nitrokey Start
Serial: FSIJ-1.2.19-5D271572
Revision: RTM.13
Config: *:*:8e82
Sys: 3.0
Board: NITROKEY-START-G
device can now be safely removed from the USB slot
final device strings: [ 'name': '', 'Vendor': 'Nitrokey', 'Product': 'Nitrokey Start', 'Serial': 'FSIJ-1.2.19-5D271572', 'Revision': 'RTM.13', 'Config': '*:*:8e82', 'Sys': '3.0', 'Board': 'NITROKEY-START-G' ]
finishing session 2023-03-16 21:49:07.371291
Log saved to: /tmp/nitropy.log.gc5753a8
jas@kaka:~$
jas@kaka:~$ nitropy start list
Command line tool to interact with Nitrokey devices 0.4.34
:: 'Nitrokey Start' keys:
FSIJ-1.2.19-5D271572: Nitrokey Nitrokey Start (RTM.13)
jas@kaka:~$
Before importing the master key to this device, it should be configured. Note the commands in the beginning to make sure scdaemon/pcscd is not running because they may have cached state from earlier cards. Change PIN code as you like after this, my experience with Gnuk was that the Admin PIN had to be changed first, then you import the key, and then you change the PIN.
jas@kaka:~$ gpg-connect-agent "SCD KILLSCD" "SCD BYE" /bye
OK
ERR 67125247 Slut p fil <GPG Agent>
jas@kaka:~$ ps auxww grep -e pcsc -e scd
jas 11651 0.0 0.0 3468 1672 pts/0 R+ 21:54 0:00 grep --color=auto -e pcsc -e scd
jas@kaka:~$ gpg --card-edit
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: [not set]
Language prefs ...: [not set]
Salutation .......:
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: off
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
gpg/card> admin
Admin commands are allowed
gpg/card> kdf-setup
gpg/card> passwd
gpg: OpenPGP card no. D276000124010200FFFE5D2715720000 detected
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
Your selection? 3
PIN changed.
1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit
Your selection? q
gpg/card> name
Cardholder's surname: Josefsson
Cardholder's given name: Simon
gpg/card> lang
Language preferences: sv
gpg/card> sex
Salutation (M = Mr., F = Ms., or space): m
gpg/card> login
Login data (account name): jas
gpg/card> url
URL to retrieve public key: https://josefsson.org/key-20190320.txt
gpg/card> forcesig
gpg/card> key-attr
Changing card key attribute for: Signature key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519
Note: There is no guarantee that the card supports the requested size.
If the key generation does not succeed, please check the
documentation of your card to see what sizes are allowed.
Changing card key attribute for: Encryption key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: cv25519
Changing card key attribute for: Authentication key
Please select what kind of key you want:
(1) RSA
(2) ECC
Your selection? 2
Please select which elliptic curve you want:
(1) Curve 25519
(4) NIST P-384
Your selection? 1
The card will now be re-configured to generate a key of type: ed25519
gpg/card>
jas@kaka:~$ gpg --card-edit
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 0
KDF setting ......: on
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]
jas@kaka:~$
Once setup, bring out your offline machine and boot it and mount your USB stick with the offline key. The paths below will be different, and this is using a somewhat unorthodox approach of working with fresh GnuPG configuration paths that I chose for the USB stick.
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$ cp -a gnupghome-backup-masterkey gnupghome-import-nitrokey-5D271572
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$ gpg --homedir $PWD/gnupghome-import-nitrokey-5D271572 --edit-key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Secret key is available.
sec ed25519/D73CF638C53C06BE
created: 2019-03-20 expired: 2019-10-22 usage: SC
trust: ultimate validity: expired
[ expired] (1). Simon Josefsson <simon@josefsson.org>
gpg> keytocard
Really move the primary key? (y/N) y
Please select where to store the key:
(1) Signature key
(3) Authentication key
Your selection? 1
sec ed25519/D73CF638C53C06BE
created: 2019-03-20 expired: 2019-10-22 usage: SC
trust: ultimate validity: expired
[ expired] (1). Simon Josefsson <simon@josefsson.org>
gpg>
Save changes? (y/N) y
jas@kaka:/media/jas/2c699cbd-b77e-4434-a0d6-0c4965864296$
At this point it is useful to confirm that the Nitrokey has the master key available and that is possible to sign statements with it, back on your regular machine:
jas@kaka:~$ gpg --card-status
Reader ...........: 20A0:4211:FSIJ-1.2.19-5D271572:0
Application ID ...: D276000124010200FFFE5D2715720000
Application type .: OpenPGP
Version ..........: 2.0
Manufacturer .....: unmanaged S/N range
Serial number ....: 5D271572
Name of cardholder: Simon Josefsson
Language prefs ...: sv
Salutation .......: Mr.
URL of public key : https://josefsson.org/key-20190320.txt
Login data .......: jas
Signature PIN ....: not forced
Key attributes ...: ed25519 cv25519 ed25519
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 3 3
Signature counter : 1
KDF setting ......: on
Signature key ....: B1D2 BD13 75BE CB78 4CF4 F8C4 D73C F638 C53C 06BE
created ....: 2019-03-20 23:37:24
Encryption key....: [none]
Authentication key: [none]
General key info..: pub ed25519/D73CF638C53C06BE 2019-03-20 Simon Josefsson <simon@josefsson.org>
sec> ed25519/D73CF638C53C06BE created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 5D271572
ssb> ed25519/80260EE8A9B92B2B created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
ssb> ed25519/51722B08FE4745A2 created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
ssb> cv25519/02923D7EE76EBD60 created: 2019-03-20 expires: 2023-09-19
card-no: FFFE 42315277
jas@kaka:~$ echo foo gpg -a --sign gpg --verify
gpg: Signature made Thu Mar 16 22:11:02 2023 CET
gpg: using EDDSA key B1D2BD1375BECB784CF4F8C4D73CF638C53C06BE
gpg: Good signature from "Simon Josefsson <simon@josefsson.org>" [ultimate]
jas@kaka:~$
Finally to retrieve and sign a key, for example Andre Heinecke s that I could confirm the OpenPGP key identifier from his business card.
jas@kaka:~$ gpg --locate-external-keys aheinecke@gnupg.com
gpg: key 1FDF723CF462B6B1: public key "Andre Heinecke <aheinecke@gnupg.com>" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 2 signed: 7 trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1 valid: 7 signed: 64 trust: 7-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2023-05-26
pub rsa3072 2015-12-08 [SC] [expires: 2025-12-05]
94A5C9A03C2FE5CA3B095D8E1FDF723CF462B6B1
uid [ unknown] Andre Heinecke <aheinecke@gnupg.com>
sub ed25519 2017-02-13 [S]
sub ed25519 2017-02-13 [A]
sub rsa3072 2015-12-08 [E] [expires: 2025-12-05]
sub rsa3072 2015-12-08 [A] [expires: 2025-12-05]
jas@kaka:~$ gpg --edit-key "94A5C9A03C2FE5CA3B095D8E1FDF723CF462B6B1"
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
pub rsa3072/1FDF723CF462B6B1
created: 2015-12-08 expires: 2025-12-05 usage: SC
trust: unknown validity: unknown
sub ed25519/2978E9D40CBABA5C
created: 2017-02-13 expires: never usage: S
sub ed25519/DC74D901C8E2DD47
created: 2017-02-13 expires: never usage: A
The following key was revoked on 2017-02-23 by RSA key 1FDF723CF462B6B1 Andre Heinecke <aheinecke@gnupg.com>
sub cv25519/1FFE3151683260AB
created: 2017-02-13 revoked: 2017-02-23 usage: E
sub rsa3072/8CC999BDAA45C71F
created: 2015-12-08 expires: 2025-12-05 usage: E
sub rsa3072/6304A4B539CE444A
created: 2015-12-08 expires: 2025-12-05 usage: A
[ unknown] (1). Andre Heinecke <aheinecke@gnupg.com>
gpg> sign
pub rsa3072/1FDF723CF462B6B1
created: 2015-12-08 expires: 2025-12-05 usage: SC
trust: unknown validity: unknown
Primary key fingerprint: 94A5 C9A0 3C2F E5CA 3B09 5D8E 1FDF 723C F462 B6B1
Andre Heinecke <aheinecke@gnupg.com>
This key is due to expire on 2025-12-05.
Are you sure that you want to sign this key with your
key "Simon Josefsson <simon@josefsson.org>" (D73CF638C53C06BE)
Really sign? (y/N) y
gpg> quit
Save changes? (y/N) y
jas@kaka:~$
This is on my day-to-day machine, using the NitroKey Start with the offline key. No need to boot the old offline machine just to sign keys or extend expiry anymore! At FOSDEM 23 I managed to get at least one DD signature on my new key, and the Debian keyring maintainers accepted my Ed25519 key. Hopefully I can now finally let my 2014-era RSA3744 key expire in 2023-09-19 and not extend it any further. This should finish my transition to a simpler OpenPGP key setup, yay!
A new release 0.2.7 of our RcppSMC package arrived at
CRAN earlier today. It contains
several extensions added by team member (and former GSoC student) Ilya Zarubin since the last
release. We were a little slow to release those but one of those CRAN
emails forced our hand for a release now. The updated uninitialized
variable messages in clang++-16 have found a fan in Brian
Ripley, and so he sent us a note. And as the issue was trivially
reproducible with clang++-15 here too I had it fixed in no
time. And both changes taken together form the incremental 0.2.7
release.
RcppSMC
provides Rcpp-based bindings to R for the Sequential Monte Carlo
Template Classes (SMCTC) by Adam Johansen described in his JSS article.
Sequential Monte Carlo is also referred to as Particle Filter
in some contexts. The package now also features the Google Summer of Code
work by Leah South in 2017, and by Ilya Zarubin in 2021.
The release is summarized below.
Changes in RcppSMC
version 0.2.7 (2023-03-22)
Extensive extensions for conditional SMC and resample, updated
hello_world example, added skeleton function for easier
package creation (Ilya in #67,#72)
Core Python Packages, by Stefano Rivera
Just before the freeze, pip added support for PEP-668.
This is a scheme devised by Debian with other distributions and the Python
Packaging Authority, to allow distributors to mark Python installations as
being managed by a distribution package manager. When this EXTERNALLY-MANAGED
flag is present, installers like pip will refuse to install packages outside
a virtual environment. This protects users from breaking unrelated software
on their systems, when installing packages with pip or similar tools. Stefano
quickly got this version of pip into the archive, marked Debian s Python
interpreters as EXTERNALLY-MANAGED, and worked with the upstream to add a
mechanism to allow users to override the restriction. Debian bookworm will
likely be the first distro release to implement this change.
The transition from Python 3.10 to 3.11 was one of the last to complete before
the bookworm freeze (as 3.11 only released at the end of October 2022). Stefano
helped port some Python packages to 3.11, in January, and also kicked off the
final transition to remove Python 3.10 support.
Stefano did a big round of bug triage in the cPython interpreter (and related)
packages, applying some provided patches, and fixing some long-standing minor
bugs in the packaging.
To allow Debian packages to more accurately reflect upstream-specified
dependencies that only apply under specific Python interpreter versions, in the
future, Stefano
added more metadata
to the python3 binary package.
Python s unittest runner would successfully exit with 0 passed tests, if it
couldn t find any tests. This means that configuration / layout changes can
cause test failures to go unnoticed, because the tests aren t being run any
more in Debian packages. Stefano
proposed a change to Python
3.12 to change this behavior and treat 0 tests as a kind of failure.
debvm, by Helmut Grohne
With support from Johannes Schauer Marin Rodrigues, and Jochen Sprickerhof,
Helmut Grohne wrote debvm, a tool for
quickly creating and running Debian virtual machine images for various
architectures and Debian and Ubuntu releases. This is meant for development
and testing purposes and has already identified a number of bugs in e.g.
fakechroot (#1029490), Linux
(#1029270), and runit
(#1028181).
Rails 6 and Redmine 5 available in bullseye-backports, by Utkarsh Gupta
Bullseye users can now upgrade to the latest 6.1 branch of Rails, v6.1.7, and
the latest Redmine version, v5.0.4. The Ruby team received numerous requests
to backport the latest version of Rails and Redmine, especially since there was
no redmine shipped in the bullseye release itself. So this is big news for all
users as we ve not only successfully backported both the packages, but also
fixed all the CVEs and RC bugs in the process!
This work was sponsored by Entrouvert.
Patches metadata in the Package Tracker, by Rapha l Hertzog
Building on the great Ultimate Debian Database work of Lucas Nussbaum and on
his suggestion, Rapha l enhanced the
Debian Package Tracker to display action items
when the patches metadata indicate that some patches were not forwarded
upstream, or when the metadata were invalid. One can now also browse the
patches metadata from the Links panel on the right.
Fixed kernel bug that broke debian-installer on computers with Mediatek wifi devices, by Helmut Grohne
As part of our regular work on Kali Linux for OffSec,
they funded Helmut s work to fix the MT7921e driver. When being loaded without
firmware available, it would not register itself, but upon module release it
would unregister itself causing a kernel oops.
This was commonly observed in Kali Linux when reloading the module to add
firmware. Helmut Grohne identified the cause and
sent a patch, a
different variant of which is now
heading into Linux
and available from Kali Linux.
Printing in Debian, by Thorsten Alteholz
There are about 40 packages in Debian that take care of sending output to
printers, scan documents, or even send documents to fax machines. In the light
of the upcoming/already ongoing freeze, these packages had to be updated to
the latest version and bugs had to be fixed. Basically this applies to large
packages like cups, cups-filters, hplip but also the smaller ones that
shouldn t be neglected. All in all Thorsten uploaded 13 packages with new
upstream versions or improved packaging and could resolve 14 bugs. Further
triaging led to 35 bugs that could be closed, either because they were already
fixed and not closed in an earlier upload or they could not be reproduced with
current software versions.
There is also work to do to prepare for the future. Historically, printing on
Linux required finding a PPD file for your printer and finding some software
that is able to render your documents with this PPD. These days, driverless
printing is becoming more common and the use of PPD files has decreased.
In the upcoming version 3.0 of cups, PPD files are no longer supported and so
called printer applications need to be used. In order not to lose the ability
to print documents, this big transition needs to be carefully planned. This
started in the beginning of 2023 and will hopefully be finished with the
release of Debian Trixie. More information can be found in
this Debian Printing Wiki article.
In preparation for this transition Thorsten created three new packages.
Yade update, by Anton Gladky
Last month, Anton updated the yade package to the newest 2023.02a version,
which includes new features.
Yade is a software package for discrete element method (DEM) simulations, which
are widely used in scientific and engineering fields for the simulation of
granular systems. Yade is an open-source project that is being used worldwide
for different tasks, such as geomechanics, civil engineering, mining, and
materials science.
The Yade package in Debian supports different precision levels for its
simulations. This means that researchers and engineers can select the needed
precision level without recompiling the package, saving time and effort.
Miscellaneous contributions
Helmut Grohne continues to improve cross building (mostly Qt) and
architecture bootstrap (mostly loong64 and musl).
The Framework is a 13.5" laptop body with swappable parts, which
makes it somewhat future-proof and certainly easily repairable,
scoring an "exceedingly rare" 10/10 score from ifixit.com.
There are two generations of the laptop's main board (both compatible
with the same body): the Intel 11th and 12th gen chipsets.
I have received my Framework, 12th generation "DIY", device in late
September 2022 and will update this page as I go along in the process
of ordering, burning-in, setting up and using the device over the
years.
Overall, the Framework is a good laptop. I like the keyboard, the
touch pad, the expansion cards. Clearly there's been some good work
done on industrial design, and it's the most repairable laptop I've
had in years. Time will tell, but it looks sturdy enough to survive me
many years as well.
This is also one of the most powerful devices I ever lay my hands
on. I have managed, remotely, more powerful servers, but this is the
fastest computer I have ever owned, and it fits in this tiny case. It
is an amazing machine.
On the downside, there's a bit of proprietary firmware required (WiFi,
Bluetooth, some graphics) and the Framework ships with a proprietary
BIOS, with currently no Coreboot support. Expect to need the
latest kernel, firmware, and hacking around a bunch of things to get
resolution and keybindings working right.
Like others, I have first found significant power management issues,
but many issues can actually be solved with some configuration. Some
of the expansion ports (HDMI, DP, MicroSD, and SSD) use power when
idle, so don't expect week-long suspend, or "full day" battery while
those are plugged in.
Finally, the expansion ports are nice, but there's only four of
them. If you plan to have a two-monitor setup, you're likely going to
need a dock.
Read on for the detailed review. For context, I'm moving from the
Purism Librem 13v4 because it
basically exploded on me. I
had, in the meantime, reverted back to an old ThinkPad X220, so I
sometimes compare the Framework with that venerable laptop as well.
This blog post has been maturing for months now. It started in
September 2022 and I declared it completed in March 2023. It's the
longest single article on this entire website, currently clocking at
about 13,000 words. It will take an average reader a full hour to go
through this thing, so I don't expect anyone to actually do
that. This introduction should be good enough for most people, read
the first section if you intend to actually buy a Framework. Jump
around the table of contents as you see fit for after you did buy the
laptop, as it might include some crucial hints on how to make it work
best for you, especially on (Debian) Linux.
Advice for buyers
Those are things I wish I would have known before buying:
consider buying 4 USB-C expansion cards, or at least a mix of 4
USB-A or USB-C cards, as they use less power than other cards and
you do want to fill those expansion slots otherwise they snag
around and feel insecure
you will likely need a dock or at least a USB hub if you want a
two-monitor setup, otherwise you'll run out of ports
you have to do some serious tuning to get proper (10h+ idle, 10
days suspend) power savings
in particular, beware that the HDMI, DisplayPort and
particularly the SSD and MicroSD cards take a significant amount
power, even when sleeping, up to 2-6W for the latter two
beware that the MicroSD card is what it says: Micro, normal SD
cards won't fit, and while there might be full sized one
eventually, it's currently only at the prototyping stage
Current status
I have the framework! It's setup with a fresh new Debian bookworm
installation. I've ran through a large number of tests and burn in.
I have decided to use the Framework as my daily driver, and had to buy
a USB-C dock to get my two monitors
connected, which was own adventure.
Update: Framework just (2023-03-23) just announced a whole bunch of
new stuff:
The recording is available in this video and it's not your
typical keynote. It starts ~25 minutes late, audio is crap, lightning
and camera are crap, clapping seems to be from whatever staff they
managed to get together in a room, decor is bizarre, colors are
shit. It's amazing.
Specifications
Those are the specifications of the 12th gen, in general terms. Your
build will of course vary according to your needs.
CPU: i5-1240P, i7-1260P, or i7-1280P (Up to 4.4-4.8 GHz, 4+8
cores), Iris Xe graphics
4 x USB-C user-selectable expansion ports, including
USB-C
USB-A
HDMI
DP
Ethernet
MicroSD
250-1000GB SSD
3.5mm combo headphone jack
Kill switches for microphone and camera
Battery: 55Wh
Camera: 1080p 60fps
Biometrics: Fingerprint Reader
Backlit keyboard
Power Adapter: 60W USB-C (or bring your own)
ships with a screwdriver/spludger
1 year warranty
base price: 1000$CAD, but doesn't give you much, typical builds
around 1500-2000$CAD
Actual build
This is the actual build I ordered. Amounts in CAD. (1CAD =
~0.75EUR/USD.)
Base configuration
CPU: Intel Core i5-1240P (AKA Alder Lake P 8 4.4GHz
P-threads, 8 3.2GHz E-threads, 16 total, 28-64W), 1079$
Memory: 16GB (1 x 16GB) DDR4-3200, 104$
Customization
Keyboard: US English, included
Expansion Cards
2 USB-C $24
3 USB-A $36
2 HDMI $50
1 DP $50
1 MicroSD $25
1 Storage 1TB $199
Sub-total: 384$
Accessories
Power Adapter - US/Canada $64.00
Total
Before tax: 1606$
After tax and duties: 1847$
Free shipping
Quick evaluation
This is basically the TL;DR: here, just focusing on broad pros/cons of
the laptop.
Pros
easily repairable (complete with QR codes pointing to repair
guides!), the 11th gen received a 10/10 score from
ifixit.com, which they call "exceedingly rare", the 12th gen
has a similar hardware design and would probably rate similarly
replaceable motherboard!!! can be reused as a NUC-like device, with a
3d-printed case, 12th gen board can be bought standalone and
retrofitted into an 11th gen case
not a passing fad: they made a first laptop with the 11th gen Intel
chipset in 2021, and a second motherboard with the 12th Intel
chipset in 2022
four modular USB-C ports which can fit HDMI, USB-C (pass-through,
can provide power on both sides), USB-A, DisplayPort, MicroSD,
external storage (250GB, 1TB), active modding community
nice power led indicating power level (charging, charged, etc) when
plugged
they used to have some difficulty keeping up with the orders: first
two batches shipped, third batch sold out, fourth batch should have
shipped in October 2021. they generally seem to keep up with
shipping. update (august 2022): they rolled out a second line of
laptops (12th gen), first batch shipped, second batch shipped
late, September 2022 batch was generally on time, see this
spreadsheet for a crowdsourced effort to track those
supply chain issues seem to be under control as of early 2023. I
got the Ethernet expansion card shipped within a week.
compared to my previous laptop (Purism Librem
13v4), it feels strangely
bulkier and heavier; it's actually lighter than the purism (1.3kg
vs 1.4kg) and thinner (15.85mm vs 18mm) but the design of the
Purism laptop (tapered edges) makes it feel thinner
no space for a 2.5" drive
rather bright LED around power button, but can be dimmed in the
BIOS (not low enough to my taste) I got used to it
fan quiet when idle, but can be noisy when running, for example if
you max a CPU for a while
battery described as "mediocre" by Ars Technica (above), confirmed
poor in my tests (see below)
no RJ-45 port, and attempts at designing ones are failing
because the modular plugs are too thin to fit (according to Linux
After Dark), so unlikely to have one in the future
Update: they cracked that nut and ship an 2.5 gbps Ethernet
expansion card with a realtek chipset, without any
firmware blob
a bit pricey for the performance, especially when compared to the
competition (e.g. Dell XPS, Apple M1)
12th gen Intel has glitchy graphics, seems like Intel hasn't fully
landed proper Linux support for that chipset yet
Initial hardware setup
A breeze.
Accessing the board
The internals are accessed through five TorX screws, but there's a nice
screwdriver/spudger that works well enough. The screws actually hold in
place so you can't even lose them.
The first setup is a bit counter-intuitive coming from the Librem
laptop, as I expected the back cover to lift and give me access to the
internals. But instead the screws is release the keyboard and touch
pad assembly, so you actually need to flip the laptop back upright and
lift the assembly off to get access to the internals. Kind of
scary.
I also actually unplugged a connector in lifting the assembly because
I lifted it towards the monitor, while you actually need to lift it
to the right. Thankfully, the connector didn't break, it just
snapped off and I could plug it back in, no harm done.
Once there, everything is well indicated, with QR codes all over the
place supposedly leading to online instructions.
Bad QR codes
Unfortunately, the QR codes I tested (in the expansion card slot, the
memory slot and CPU slots) did not actually work so I wonder how
useful those actually are.
After all, they need to point to something and that means a URL, a
running website that will answer those requests forever. I bet those
will break sooner than later and in fact, as far as I can tell, they
just don't work at all. I prefer the approach taken by the MNT reform
here which designed (with the 100 rabbits folks) an actual paper
handbook (PDF).
The first QR code that's immediately visible from the back of the
laptop, in an expansion cord slot, is a 404. It seems to be some
serial number URL, but I can't actually tell because, well, the page
is a 404.
I was expecting that bar code to lead me to an introduction page,
something like "how to setup your Framework laptop". Support actually
confirmed that it should point a quickstart guide. But in a
bizarre twist, they somehow sent me the URL with the plus (+) signs
escaped, like this:
(They have also "let the team know about this for feedback and help
resolve the problem with the link" which is a support code word for
"ha-ha! nope! not my problem right now!" Trust me, I know, my own
code word is "can you please make a ticket?")
Seating disks and memory
The "DIY" kit doesn't actually have that much of a setup. If you
bought RAM, it's shipped outside the laptop in a little plastic case,
so you just seat it in as usual.
Then you insert your NVMe drive, and, if that's your fancy, you also
install your own mPCI WiFi card. If you ordered one (which was my
case), it's pre-installed.
Closing the laptop is also kind of amazing, because the keyboard
assembly snaps into place with magnets. I have actually used the
laptop with the keyboard unscrewed as I was putting the drives in and
out, and it actually works fine (and will probably void your warranty,
so don't do that). (But you can.) (But don't, really.)
Hardware review
Keyboard and touch pad
The keyboard feels nice, for a laptop. I'm used to mechanical keyboard
and I'm rather violent with those poor things. Yet the key travel is
nice and it's clickety enough that I don't feel too disoriented.
At first, I felt the keyboard as being more laggy than my normal
workstation setup, but it turned out this was a graphics driver
issues. After enabling a composition manager, everything feels snappy.
The touch pad feels good. The double-finger scroll works well enough,
and I don't have to wonder too much where the middle button is, it
just works.
Taps don't work, out of the box: that needs to be enabled in Xorg,
with something like this:
But be aware that once you enable that tapping, you'll need to deal
with palm detection... So I have not actually enabled this in the end.
Power button
The power button is a little dangerous. It's quite easy to hit, as
it's right next to one expansion card where you are likely to plug in
a cable power. And because the expansion cards are kind of hard to
remove, you might squeeze the laptop (and the power key) when trying
to remove the expansion card next to the power button.
So obviously, don't do that. But that's not very helpful.
An alternative is to make the power button do something else. With
systemd-managed systems, it's actually quite easy. Add a
HandlePowerKey stanza to (say)
/etc/systemd/logind.conf.d/power-suspends.conf:
And the power button will suspend! Long-press to power off doesn't
actually work as the laptop immediately suspends...
Note that there's probably half a dozen other ways of doing this,
see this, this, or that.
Special keybindings
There is a series of "hidden" (as in: not labeled on the key)
keybindings related to the fn keybinding that I actually
find quite useful.
Key
Equivalent
Effect
Command
p
Pause
lock screen
xset s activate
b
Break
?
?
k
ScrLk
switch keyboard layout
N/A
It looks like those are defined in the microcontroller so it
would be possible to add some. For example, the SysRq key
is almost bound to fns in there.
Note that most other shortcuts like this are clearly documented
(volume, brightness, etc). One key that's less obvious is
F12 that only has the Framework logo on it. That actually
calls the keysym XF86AudioMedia which, interestingly, does
absolutely nothing here. By default, on Windows, it opens your
browser to the Framework website and, on Linux, your "default
media player".
The keyboard backlight can be cycled with fn-space. The
dimmer version is dim enough, and the keybinding is easy to find in
the dark.
A skinny elephant would be performed with altPrtScr (above F11) KEY, so for
example altfnF11b
should do a hard reset. This comment suggests you need to hold
the fnonly if "function lock" is on, but that's
actually the opposite of my experience.
Out of the box, some of the fn keys don't work. Mute,
volume up/down, brightness, monitor changes, and the airplane mode key
all do basically nothing. They don't send proper keysyms to Xorg at
all.
This is a known problem and it's related to the fact that the
laptop has light sensors to adjust the brightness
automatically. Somehow some of those keys (e.g. the brightness
controls) are supposed to show up as a different input device, but
don't seem to work correctly. It seems like the solution is for the
Framework team to write a driver specifically for this, but so far no
progress since July 2022.
In the meantime, the fancy functionality can be supposedly disabled with:
echo 'blacklist hid_sensor_hub' sudo tee /etc/modprobe.d/framework-als-blacklist.conf
Kill switches
The Framework has two "kill switches": one for the camera and the
other for the microphone. The camera one actually disconnects the USB
device when turned off, and the mic one seems to cut the circuit. It
doesn't show up as muted, it just stops feeding the sound.
Both kill switches are around the main camera, on top of the monitor,
and quite discreet. Then turn "red" when enabled (i.e. "red" means
"turned off").
Monitor
The monitor looks pretty good to my untrained eyes. I have yet to do
photography work on it, but some photos I looked at look sharp and the
colors are bright and lively. The blacks are dark and the screen is
bright.
I have yet to use it in full sunlight.
The dimmed light is very dim, which I like.
Screen backlight
I bind brightness keys to xbacklight in i3, but out of the box I get
this error:
sep 29 22:09:14 angela i3[5661]: No outputs have backlight property
It just requires this blob in /etc/X11/xorg.conf.d/backlight.conf:
This way I can control the actual backlight power with the brightness
keys, and they do significantly reduce power usage.
Multiple monitor support
I have been able to hook up my two old monitors to the HDMI and
DisplayPort expansion cards on the laptop. The lid closes without
suspending the machine, and everything works great.
I actually run out of ports, even with a 4-port USB-A hub, which gives
me a total of 7 ports:
power (USB-C)
monitor 1 (DisplayPort)
monitor 2 (HDMI)
USB-A hub, which adds:
keyboard (USB-A)
mouse (USB-A)
Yubikey
external sound card
Now the latter, I might be able to get rid of if I switch to a
combo-jack headset, which I do have (and still need to test).
But still, this is a problem. I'll probably need a powered USB-C dock
and better monitors, possibly with some Thunderbolt chaining, to
save yet more ports.
But that means more money into this setup, argh. And figuring out my
monitor situation is the kind of thing I'm not that big
of a fan of. And neither is shopping for USB-C (or is it Thunderbolt?)
hubs.
My normal autorandr setup doesn't work: I have tried saving a
profile and it doesn't get autodetected, so I also first need to do:
autorandr -l framework-external-dual-lg-acer
The magic:
autorandr -l horizontal
... also works well.
The worst problem with those monitors right now is that they have a
radically smaller resolution than the main screen on the laptop, which
means I need to reset the font scaling to normal every time I switch
back and forth between those monitors and the laptop, which means I
actually need to do this:
Expansion ports
I ordered a total of 10 expansion ports.
I did manage to initialize the 1TB drive as an encrypted storage,
mostly to keep photos as this is something that takes a massive amount
of space (500GB and counting) and that I (unfortunately) don't work on
very often (but still carry around).
The expansion ports are fancy and nice, but not actually that
convenient. They're a bit hard to take out: you really need to crimp
your fingernails on there and pull hard to take them out. There's a
little button next to them to release, I think, but at first it feels
a little scary to pull those pucks out of there. You get used to it
though, and it's one of those things you can do without looking
eventually.
There's only four expansion ports. Once you have two monitors, the
drive, and power plugged in, bam, you're out of ports; there's nowhere
to plug my Yubikey. So if this is going to be my daily driver, with a
dual monitor setup, I will need a dock, which means more crap firmware
and uncertainty, which isn't great. There are actually plans to make a
dual-USB card, but that is blocked on designing an actual
board for this.
I can't wait to see more expansion ports produced. There's a ethernet
expansion card which quickly went out of stock basically the day
it was announced, but was eventually restocked.
I would like to see a proper SD-card reader. There's a MicroSD card
reader, but that obviously doesn't work for normal SD cards, which
would be more broadly compatible anyways (because you can have a
MicroSD to SD card adapter, but I have never heard of the
reverse). Someone actually found a SD card reader that fits and
then someone else managed to cram it in a 3D printed case, which
is kind of amazing.
Still, I really like that idea that I can carry all those little
adapters in a pouch when I travel and can basically do anything I
want. It does mean I need to shuffle through them to find the right
one which is a little annoying. I have an elastic band to keep them
lined up so that all the ports show the same side, to make it easier
to find the right one. But that quickly gets undone and instead I have
a pouch full of expansion cards.
Another awesome thing with the expansion cards is that they don't just
work on the laptop: anything that takes USB-C can take those cards,
which means you can use it to connect an SD card to your phone, for
backups, for example. Heck, you could even connect an external display
to your phone that way, assuming that's supported by your phone of
course (and it probably isn't).
The expansion ports do take up some power, even when idle. See the
power management section below, and particularly the power usage
tests for details.
USB-C charging
One thing that is really a game changer for me is USB-C charging. It's
hard to overstate how convenient this is. I often have a USB-C cable
lying around to charge my phone, and I can just grab that thing and
pop it in my laptop. And while it will obviously not charge as fast as
the provided charger, it will stop draining the battery at least.
(As I wrote this, I had the laptop plugged in the Samsung charger that
came with a phone, and it was telling me it would take 6 hours to
charge the remaining 15%. With the provided charger, that flew down to
15 minutes. Similarly, I can power the laptop from the power grommet
on my desk, reducing clutter as I have that single wire out there
instead of the bulky power adapter.)
I also really like the idea that I can charge my laptop with a power
bank or, heck, with my phone, if push comes to shove. (And
vice-versa!)
This is awesome. And it works from any of the expansion ports, of
course. There's a little led next to the expansion ports as well,
which indicate the charge status:
red/amber: charging
white: charged
off: unplugged
I couldn't find documentation about this, but the forum
answered.
This is something of a recurring theme with the Framework. While it
has a good knowledge base and repair/setup guides (and the
forum is awesome) but it doesn't have a good "owner manual" that
shows you the different parts of the laptop and what they do. Again,
something the MNT reform did well.
Another thing that people are asking about is an external sleep
indicator: because the power LED is on the main keyboard assembly,
you don't actually see whether the device is active or not when the
lid is closed.
Finally, I wondered what happens when you plug in multiple power
sources and it turns out the charge controller is actually pretty
smart: it will pick the best power source and use it. The only
downside is it can't use multiple power sources, but that seems like
a bit much to ask.
Multimedia and other devices
Those things also work:
webcam: splendid, best webcam I've ever had (but my standards are
really low)
onboard mic: works well, good gain (maybe a bit much)
onboard speakers: sound okay, a little metal-ish, loud enough to be
annoying, see this thread for benchmarks, apparently pretty
good speakers
Combo jack mic tests
The Framework laptop ships with a combo jack on the left side, which
allows you to plug in a CTIA (source) headset. In human
terms, it's a device that has both a stereo output and a mono input,
typically a headset or ear buds with a microphone somewhere.
It works, which is better than the Purism (which only had audio
out), but is on par for the course for that kind of onboard
hardware. Because of electrical interference, such sound cards very
often get lots of noise from the board.
With a Jabra Evolve 40, the built-in USB sound card generates
basically zero noise on silence (invisible down to -60dB in Audacity)
while plugging it in directly generates a solid -30dB hiss. There is
a noise-reduction system in that sound card, but the difference is
still quite striking.
On a comparable setup (curie, a 2017 Intel NUC), there is
also a his with the Jabra headset, but it's quieter, more in the order
of -40/-50 dB, a noticeable difference. Interestingly, testing with my
Mee Audio Pro M6 earbuds leads to a little more hiss on curie, more on
the -35/-40 dB range, close to the Framework.
Also note that another sound card, the Antlion USB adapter that comes
with the ModMic 4, also gives me pretty close to silence on a quiet
recording, picking up less than -50dB of background noise. It's
actually probably picking up the fans in the office, which do make
audible noises.
In other words, the hiss of the sound card built in the Framework
laptop is so loud that it makes more noise than the quiet fans in the
office. Or, another way to put it is that two USB sound cards (the
Jabra and the Antlion) are able to pick up ambient noise in my office
but not the Framework laptop.
See also my audio page.
Performance tests
Compiling Linux 5.19.11
On a single core, compiling the Debian version of the Linux kernel
takes around 100 minutes:
I had to plug the normal power supply after a few minutes because
battery would actually run out using my desk's power grommet (34
watts).
During compilation, fans were spinning really hard, quite noisy, but
not painfully so.
The laptop was sucking 55 watts of power, steadily:
Time User Nice Sys Idle IO Run Ctxt/s IRQ/s Fork Exec Exit Watts
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Average 87.9 0.0 10.7 1.4 0.1 17.8 6583.6 5054.3 233.0 223.9 233.1 55.96
GeoMean 87.9 0.0 10.6 1.2 0.0 17.6 6427.8 5048.1 227.6 218.7 227.7 55.96
StdDev 1.4 0.0 1.2 0.6 0.2 3.0 1436.8 255.5 50.0 47.5 49.7 0.20
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Minimum 85.0 0.0 7.8 0.5 0.0 13.0 3594.0 4638.0 117.0 111.0 120.0 55.52
Maximum 90.8 0.0 12.9 3.5 0.8 38.0 10174.0 5901.0 374.0 362.0 375.0 56.41
-------- ----- ----- ----- ----- ----- ---- ------ ------ ---- ---- ---- ------
Summary:
CPU: 55.96 Watts on average with standard deviation 0.20
Note: power read from RAPL domains: package-0, uncore, package-0, core, psys.
These readings do not cover all the hardware in this device.
memtest86+
I ran Memtest86+ v6.00b3. It shows something like this:
Software setup
Once I had everything in the hardware setup, I figured, voil , I'm
done, I'm just going to boot this beautiful machine and I can get back
to work.
I don't understand why I am so na ve some times. It's mind boggling.
Obviously, it didn't happen that way at all, and I spent the best of
the three following days tinkering with the laptop.
Secure boot and EFI
First, I couldn't boot off of the NVMe drive I transferred from the
previous laptop (the Purism) and the
BIOS was not very helpful: it was just complaining about not finding
any boot device, without dropping me in the real BIOS.
At first, I thought it was a problem with my NVMe drive, because it's
not listed in the compatible SSD drives from upstream. But I
figured out how to enter BIOS (press F2 manically, of
course), which showed the NVMe drive was actually detected. It just
didn't boot, because it was an old (2010!!) Debian install without
EFI.
So from there, I disabled secure boot, and booted a grml image to
try to recover. And by "boot" I mean, I managed to get to the grml
boot loader which promptly failed to load its own root file system
somehow. I still have to investigate exactly what happened there, but
it failed some time after the initrd load with:
Unable to find medium containing a live file system
This, it turns out, was fixed in Debian lately, so a daily GRML
build will not have this problems. The upcoming 2022 release
(likely 2022.10 or 2022.11) will also get the fix.
I did manage to boot the development version of the Debian
installer which was a surprisingly good experience: it mounted the
encrypted drives and did everything pretty smoothly. It even offered
me to reinstall the boot loader, but that ultimately (and correctly, as
it turns out) failed because I didn't have a /boot/efi partition.
At this point, I realized there was no easy way out of this, and I
just proceeded to completely reinstall Debian. I had a spare NVMe
drive lying around (backups FTW!) so I just swapped that in, rebooted
in the Debian installer, and did a clean install. I wanted to switch
to bookworm anyways, so I guess that's done too.
Storage limitations
Another thing that happened during setup is that I tried to copy over
the internal 2.5" SSD drive from the Purism to the Framework 1TB
expansion card. There's no 2.5" slot in the new laptop, so that's
pretty much the only option for storage expansion.
I was tired and did something wrong. I ended up wiping the partition
table on the original 2.5" drive.
Oops.
It might be recoverable, but just restoring the partition table
didn't work either, so I'm not sure how I recover the data
there. Normally, everything on my laptops and workstations is designed
to be disposable, so that wasn't that big of a problem. I did manage
to recover most of the data thanks to git-annexreinit, but
that was a little hairy.
Bootstrapping Puppet
Once I had some networking, I had to install all the packages I
needed. The time I spent setting up my workstations with Puppet has
finally paid off. What I actually did was to restore two critical
directories:
/etc/ssh
/var/lib/puppet
So that I would keep the previous machine's identity. That way I could
contact the Puppet server and install whatever was missing. I used my
Puppet optimization
trick to do a batch
install and then I had a good base setup, although not exactly as it
was before. 1700 packages were installed manually on angela before
the reinstall, and not in Puppet.
I did not inspect each one individually, but I did go through /etc
and copied over more SSH keys, for backups and SMTP over SSH.
LVFS support
It looks like there's support for the (de-facto) standard LVFS
firmware update system. At least I was able to update the UEFI
firmware with a simple:
Those instructions come from the beta forum post. I performed the
BIOS update on 2023-01-16T16:00-0500.
Resolution tweaks
The Framework laptop resolution (2256px X 1504px) is big enough to
give you a pretty small font size, so welcome to the marvelous world
of "scaling".
The Debian wiki page has a few tricks for this.
Console
This will make the console and grub fonts more readable:
Xorg
Adding this to your .Xresources will make everything look much bigger:
! 1.5*96
Xft.dpi: 144
Apparently, some of this can also help:
! These might also be useful depending on your monitor and personal preference:
Xft.autohint: 0
Xft.lcdfilter: lcddefault
Xft.hintstyle: hintfull
Xft.hinting: 1
Xft.antialias: 1
Xft.rgba: rgb
It my experience it also makes things look a little fuzzier, which is
frustrating because you have this awesome monitor but everything looks
out of focus. Just bumping Xft.dpi by a 1.5 factor looks good to me.
The Debian Wiki has a page on HiDPI, but it's not as good as the
Arch Wiki, where the above blurb comes from. I am not using the
latter because I suspect it's causing some of the "fuzziness".
TODO: find the equivalent of this GNOME hack in i3? (gsettings set
org.gnome.mutter experimental-features
"['scale-monitor-framebuffer']"), taken from this Framework
guide
Issues
BIOS configuration
The Framework BIOS has some minor issues. One issue I personally
encountered is that I had disabled Quick boot and Quiet boot in
the BIOS to diagnose the above boot issues. This, in turn, triggers a
bug where the BIOS boot manager (F12) would just hang
completely. It would also fail to boot from an external USB drive.
The current fix (as of BIOS 3.03) is to re-enable both Quick
boot and Quiet boot. Presumably this is something that will get
fixed in a future BIOS update.
Note that the following keybindings are active in the BIOS POST
check:
Key
Meaning
F2
Enter BIOS setup menu
F12
Enter BIOS boot manager
Delete
Enter BIOS setup menu
WiFi compatibility issues
I couldn't make WiFi work at first. Obviously, the default Debian
installer doesn't ship with proprietary firmware (although that might
change soon) so the WiFi card didn't work out of the box. But even
after copying the firmware through a USB stick, I couldn't quite
manage to find the right combination of ip/iw/wpa-supplicant
(yes, after repeatedly copying a bunch more packages over to get
those bootstrapped). (Next time I should probably try something like
this post.)
Thankfully, I had a little USB-C dongle with a RJ-45 jack lying
around. That also required a firmware blob, but it was a single
package to copy over, and with that loaded, I had network.
Eventually, I did managed to make WiFi work; the problem was more on
the side of "I forgot how to configure a WPA network by hand from the
commandline" than anything else. NetworkManager worked fine and got
WiFi working correctly.
Note that this is with Debian bookworm, which has the 5.19 Linux
kernel, and with the firmware-nonfree (firmware-iwlwifi,
specifically) package.
Battery life
I was having between about 7 hours of battery on the Purism Librem
13v4, and that's after a year or two of battery life. Now, I still
have about 7 hours of battery life, which is nicer than my old
ThinkPad X220 (20 minutes!) but really, it's not that good for a new
generation laptop. The 12th generation Intel chipset probably improved
things compared to the previous one Framework laptop, but I don't have
a 11th gen Framework to compare with).
(Note that those are estimates from my status bar, not wall clock
measurements. They should still be comparable between the Purism and
Framework, that said.)
The battery life doesn't seem up to, say, Dell XPS 13, ThinkPad X1, and
of course not the Apple M1, where I would expect 10+ hours of battery
life out of the box.
That said, I do get those kind estimates when the machine is fully
charged and idle. In fact, when everything is quiet and nothing is
plugged in, I get dozens of hours of battery life estimated (I've
seen 25h!). So power usage fluctuates quite a bit depending on usage,
which I guess is expected.
Concretely, so far, light web browsing, reading emails and writing
notes in Emacs (e.g. this file) takes about 8W of power:
Expansion cards matter a lot in the battery life (see below for a
thorough discussion), my normal setup is 2xUSB-C and 1xUSB-A (yes,
with an empty slot, and yes, to save power).
Interestingly, playing a video in a (720p) window in a window takes up
more power (10.5W) than in full screen (9.5W) but I blame that on my
desktop setup (i3 + compton)... Not sure if mpv hits the
VA-API, maybe not in windowed mode. Similar results with 1080p,
interestingly, except the window struggles to keep up altogether. Full
screen playback takes a relatively comfortable 9.5W, which means a
solid 5h+ of playback, which is fine by me.
Fooling around the web, small edits, youtube-dl, and I'm at around 80%
battery after about an hour, with an estimated 5h left, which is a
little disappointing. I had a 7h remaining estimate before I started
goofing around Discourse, so I suspect the website is a pretty
big battery drain, actually. I see about 10-12 W, while I was probably at
half that (6-8W) just playing music with mpv in the background...
In other words, it looks like editing posts in Discourse with Firefox
takes a solid 4-6W of power. Amazing and gross.
(When writing about abusive power usage generates more power usage, is
that an heisenbug? Or schr dinbug?)
Power management
Compared to the Purism Librem 13v4, the ongoing power usage seems to
be slightly better. An anecdotal metric is that the Purism would take
800mA idle, while the more powerful Framework manages a little over
500mA as I'm typing this, fluctuating between 450 and 600mA. That is
without any active expansion card, except the storage. Those numbers
come from the output of tlp-stat -b and, unfortunately, the "ampere"
unit makes it quite hard to compare those, because voltage is not
necessarily the same between the two platforms.
TODO: i915 driver has a lot of parameters, including some about
power saving, see, again, the arch wiki, and particularly
enable_fbc=1
TL:DR; power management on the laptop is an issue, but there's various
tweaks you can make to improve it. Try:
powertop --auto-tune
apt install tlp && systemctl enable tlp
nvme.noacpi=1 mem_sleep_default=deep on the kernel command line
may help with standby power usage
keep only USB-C expansion cards plugged in, all others suck power
even when idle
consider upgrading the BIOS to latest beta (3.06 at the time of
writing), unverified power savings
latest Linux kernels (6.2) promise power savings as well
(unverified)
Update: also try to follow the official optimization guide. It
was made for Ubuntu but will probably also work for your distribution
of choice with a few tweaks. They recommend using tlpui but it's
not packaged in Debian. There is, however, a Flatpak
release. In my case, it resulted in the following diff to
tlp.conf: tlp.patch.
Background on CPU architecture
There were power problems in the 11th gen Framework laptop, according
to this report from Linux After Dark, so the issues with power
management on the Framework are not new.
The 12th generation Intel CPU (AKA "Alder Lake") is a big-little
architecture with "power-saving" and "performance" cores. There
used to be performance problems introduced by the scheduler in Linux
5.16 but those were eventually fixed in 5.18, which uses
Intel's hardware as an "intelligent, low-latency hardware-assisted
scheduler". According to Phoronix, the 5.19 release improved the
power saving, at the cost of some penalty cost. There were also patch
series to make the scheduler configurable, but it doesn't look
those have been merged as of 5.19. There was also a session about this
at the 2022 Linux Plumbers, but they stopped short of
talking more about the specific problems Linux is facing in Alder
lake:
Specifically, the kernel's energy-aware scheduling heuristics don't
work well on those CPUs. A number of features present there
complicate the energy picture; these include SMT, Intel's "turbo
boost" mode, and the CPU's internal power-management mechanisms. For
many workloads, running on an ostensibly more power-hungry Pcore can
be more efficient than using an Ecore. Time for discussion of the
problem was lacking, though, and the session came to a close.
All this to say that the 12gen Intel line shipped with this Framework
series should have better power management thanks to its
power-saving cores. And Linux has had the scheduler changes to make
use of this (but maybe is still having trouble). In any case, this
might not be the source of power management problems on my laptop,
quite the opposite.
Also note that the firmware updates for various chipsets are
supposed to improve things eventually.
On the other hand, The Verge simply declared the whole P-series
a mistake...
Attempts at improving power usage
I did try to follow some of the tips in this forum post. The
tricks powertop --auto-tune and tlp's
PCIE_ASPM_ON_BAT=powersupersave basically did nothing: I was stuck
at 10W power usage in powertop (600+mA in tlp-stat).
Apparently, I should be able to reach the C8 CPU power state (or
even C9, C10) in powertop, but I seem to be stock at
C7. (Although I'm not sure how to read that tab in powertop: in the
Core(HW) column there's only C3/C6/C7 states, and most cores are 85%
in C7 or maybe C6. But the next column over does show many CPUs in
C10 states...
As it turns out, the graphics card actually takes up a good chunk of
power unless proper power management is enabled (see below). After
tweaking this, I did manage to get down to around 7W power usage in
powertop.
Expansion cards actually do take up power, and so does the screen,
obviously. The fully-lit screen takes a solid 2-3W of power compared
to the fully dimmed screen. When removing all expansion cards and
making the laptop idle, I can spin it down to 4 watts power usage at
the moment, and an amazing 2 watts when the screen turned off.
Caveats
Abusive (10W+) power usage that I initially found could be a problem
with my desktop configuration: I have this silly status bar that
updates every second and probably causes redraws... The CPU certainly
doesn't seem to spin down below 1GHz. Also note that this is with an
actual desktop running with everything: it could very well be that
some things (I'm looking at you Signal Desktop) take up unreasonable
amount of power on their own (hello, 1W/electron, sheesh). Syncthing
and containerd (Docker!) also seem to take a good 500mW just sitting
there.
Beyond my desktop configuration, this could, of course, be a
Debian-specific problem; your favorite distribution might be better at
power management.
Idle power usage tests
Some expansion cards waste energy, even when unused. Here is a summary
of the findings from the powerstat page. I also include other
devices tested in this page for completeness:
Device
Minimum
Average
Max
Stdev
Note
Screen, 100%
2.4W
2.6W
2.8W
N/A
Screen, 1%
30mW
140mW
250mW
N/A
Backlight 1
290mW
?
?
?
fairly small, all things considered
Backlight 2
890mW
1.2W
3W?
460mW?
geometric progression
Backlight 3
1.69W
1.5W
1.8W?
390mW?
significant power use
Radios
100mW
250mW
N/A
N/A
USB-C
N/A
N/A
N/A
N/A
negligible power drain
USB-A
10mW
10mW
?
10mW
almost negligible
DisplayPort
300mW
390mW
600mW
N/A
not passive
HDMI
380mW
440mW
1W?
20mW
not passive
1TB SSD
1.65W
1.79W
2W
12mW
significant, probably higher when busy
MicroSD
1.6W
3W
6W
1.93W
highest power usage, possibly even higher when busy
Ethernet
1.69W
1.64W
1.76W
N/A
comparable to the SSD card
So it looks like all expansion cards but the USB-C ones are active,
i.e. they draw power with idle. The USB-A cards are the least concern,
sucking out 10mW, pretty much within the margin of error. But both the
DisplayPort and HDMI do take a few hundred miliwatts. It looks like
USB-A connectors have this fundamental flaw that they necessarily draw
some powers because they lack the power negotiation features of
USB-C. At least according to this post:
It seems the USB A must have power going to it all the time, that
the old USB 2 and 3 protocols, the USB C only provides power when
there is a connection. Old versus new.
Apparently, this is a problem specific to the USB-C to USB-A
adapter that ships with the Framework. Some people have actually
changed their orders to all USB-C because of this problem, but I'm
not sure the problem is as serious as claimed in the forums. I
couldn't reproduce the "one watt" power drains suggested elsewhere,
at least not repeatedly. (A previous version of this post did show
such a power drain, but it was in a less controlled test
environment than the series of more rigorous tests above.)
The worst offenders are the storage cards: the SSD drive takes at
least one watt of power and the MicroSD card seems to want to take all
the way up to 6 watts of power, both just sitting there doing
nothing. This confirms claims of 1.4W for the SSD (but not
5W) power usage found elsewhere. The former post has
instructions on how to disable the card in software. The MicroSD card
has been reported as using 2 watts, but I've seen it as high as 6
watts, which is pretty damning.
The Framework team has a beta update for the DisplayPort adapter
but currently only for Windows (LVFS technically possible, "under
investigation"). A USB-A firmware update is alsounder
investigation. It is therefore likely at least some of those power
management issues will eventually be fixed.
Note that the upcoming Ethernet card has a reported 2-8W power usage,
depending on traffic. I did my own power usage tests in
powerstat-wayland and they seem lower than 2W.
The upcoming 6.2 Linux kernel might also improve battery usage when
idle, see this Phoronix article for details, likely in early
2023.
Idle power usage tests under Wayland
Update: I redid those tests under Wayland, see powerstat-wayland
for details. The TL;DR: is that power consumption is either smaller or
similar.
Idle power usage tests, 3.06 beta BIOS
I redid the idle tests after the 3.06 beta BIOS update and ended
up with this results:
Device
Minimum
Average
Max
Stdev
Note
Baseline
1.96W
2.01W
2.11W
30mW
1 USB-C, screen off, backlight off, no radios
2 USB-C
1.95W
2.16W
3.69W
430mW
USB-C confirmed as mostly passive...
3 USB-C
1.95W
2.16W
3.69W
430mW
... although with extra stdev
1TB SSD
3.72W
3.85W
4.62W
200mW
unchanged from before upgrade
1 USB-A
1.97W
2.18W
4.02W
530mW
unchanged
2 USB-A
1.97W
2.00W
2.08W
30mW
unchanged
3 USB-A
1.94W
1.99W
2.03W
20mW
unchanged
MicroSD w/o card
3.54W
3.58W
3.71W
40mW
significant improvement! 2-3W power saving!
MicroSD w/ card
3.53W
3.72W
5.23W
370mW
new measurement! increased deviation
DisplayPort
2.28W
2.31W
2.37W
20mW
unchanged
1 HDMI
2.43W
2.69W
4.53W
460mW
unchanged
2 HDMI
2.53W
2.59W
2.67W
30mW
unchanged
External USB
3.85W
3.89W
3.94W
30mW
new result
Ethernet
3.60W
3.70W
4.91W
230mW
unchanged
Note that the table summary is different than the previous table: here
we show the absolute numbers while the previous table was doing a
confusing attempt at showing relative (to the baseline) numbers.
Conclusion: the 3.06 BIOS update did not significantly change idle
power usage stats except for the MicroSD card which has
significantly improved.
The new "external USB" test is also interesting: it shows how the
provided 1TB SSD card performs (admirably) compared to existing
devices. The other new result is the MicroSD card with a card which,
interestingly, uses less power than the 1TB SSD drive.
That's 8mAh per 10 minutes (and 2 seconds), or 48mA, or, with this
battery, about 127 hours or roughly 5 days of standby. Not bad!
In comparison, here is my really old x220, before:
sep 29 22:13:54 emma systemd-sleep[176315]: /sys/class/power_supply/BAT0/energy_now = 5070 [mWh]
... after:
sep 29 22:23:54 emma systemd-sleep[176486]: /sys/class/power_supply/BAT0/energy_now = 4980 [mWh]
... which is 90 mwH in 10 minutes, or a whopping 540mA, which was
possibly okay when this battery was new (62000 mAh, so about 100
hours, or about 5 days), but this battery is almost dead and has
only 5210 mAh when full, so only 10 hours standby.
And here is the Framework performing a similar test, before:
... which is 49mAh in a little over 10 minutes (and 4 seconds), or
292mA, much more than the Purism, but half of the X220. At this rate,
the battery would last on standby only 12 hours!! That is pretty
bad.
Note that this was done with the following expansion cards:
2 USB-C
1 1TB SSD drive
1 USB-A with a hub connected to it, with keyboard and LAN
Preliminary tests without the hub (over one minute) show that it
doesn't significantly affect this power consumption (300mA).
This guide also suggests booting with nvme.noacpi=1 but this
still gives me about 5mAh/min (or 300mA).
Adding mem_sleep_default=deep to the kernel command line does make a
difference. Before:
... which is 2mAh in 74 seconds, which is 97mA, brings us to a more
reasonable 36 hours, or a day and a half. It's still above the x220
power usage, and more than an order of magnitude more than the Purism
laptop. It's also far from the 0.4% promised by upstream, which
would be 14mA for the 3500mAh battery.
It should also be noted that this "deep" sleep mode is a little more
disruptive than regular sleep. As you can see by the timing, it took
more than 10 seconds for the laptop to resume, which feels a little
alarming as your banging the keyboard to bring it back to life.
You can confirm the current sleep mode with:
# cat /sys/power/mem_sleep
s2idle [deep]
In the above, deep is selected. You can change it on the fly with:
... better! 6 mAh in about 6 minutes, works out to 63.5mA, so more
than two days standby.
A longer test:
oct 01 09:22:56 angela systemd-sleep[62978]: /sys/class/power_supply/BAT1/charge_now = 3327 [mAh]
oct 01 12:47:35 angela systemd-sleep[63219]: /sys/class/power_supply/BAT1/charge_now = 3147 [mAh]
That's 180mAh in about 3.5h, 52mA! Now at 66h, or almost 3 days.
I wasn't sure why I was seeing such fluctuations in those tests, but
as it turns out, expansion card power tests show that they do
significantly affect power usage, especially the SSD drive, which can
take up to two full watts of power even when idle. I didn't control
for expansion cards in the above tests running them with whatever
card I had plugged in without paying attention so it's likely the
cause of the high power usage and fluctuations.
It might be possible to work around this problem by disabling USB
devices before suspend. TODO. See also this post.
In the meantime, I have been able to get much better suspend
performance by unplugging all modules. Then I get this result:
oct 04 11:15:38 angela systemd-sleep[257571]: /sys/class/power_supply/BAT1/charge_now = 3203 [mAh]
oct 04 15:09:32 angela systemd-sleep[257866]: /sys/class/power_supply/BAT1/charge_now = 3145 [mAh]
Which is 14.8mA! Almost exactly the number promised by Framework! With
a full battery, that means a 10 days suspend time. This is actually
pretty good, and far beyond what I was expecting when starting down
this journey.
So, once the expansion cards are unplugged, suspend power usage is
actually quite reasonable. More detailed standby tests are available
in the standby-tests page, with a summary below.
There is also some hope that the Chromebook edition
specifically designed with a specification of 14 days standby
time could bring some firmware improvements back down to the
normal line. Some of those issues were reported upstream in April
2022, but there doesn't seem to have been any progress there
since.
TODO: one final solution here is suspend-then-hibernate, which
Windows uses for this
TODO: consider implementing the S0ix sleep states , see also troubleshooting
TODO: consider https://github.com/intel/pm-graph
Standby expansion cards test results
This table is a summary of the more extensive standby-tests I have performed:
Device
Wattage
Amperage
Days
Note
baseline
0.25W
16mA
9
sleep=deep nvme.noacpi=1
s2idle
0.29W
18.9mA
~7
sleep=s2idle nvme.noacpi=1
normal nvme
0.31W
20mA
~7
sleep=s2idle without nvme.noacpi=1
1 USB-C
0.23W
15mA
~10
2 USB-C
0.23W
14.9mA
same as above
1 USB-A
0.75W
48.7mA
3
+500mW (!!) for the first USB-A card!
2 USB-A
1.11W
72mA
2
+360mW
3 USB-A
1.48W
96mA
<2
+370mW
1TB SSD
0.49W
32mA
<5
+260mW
MicroSD
0.52W
34mA
~4
+290mW
DisplayPort
0.85W
55mA
<3
+620mW (!!)
1 HDMI
0.58W
38mA
~4
+250mW
2 HDMI
0.65W
42mA
<4
+70mW
Conclusions:
USB-C cards take no extra power on suspend, possibly less
than empty slots, more testing required
USB-A cards take a lot more power on suspend
(300-500mW) than on regular idle (~10mW, almost negligible)
1TB SSD and MicroSD cards seem to take a reasonable
amount of power (260-290mW), compared to their runtime
equivalents (1-6W!)
DisplayPort takes a surprising lot of power (620mW), almost
double its average runtime usage (390mW)
HDMI cards take, surprisingly, less power (250mW) in
standby than the DP card (620mW)
and oddly, a second card adds less power usage (70mW?!) than the
first, maybe a circuit is used by both?
Standby expansion cards test results, 3.06 beta BIOS
Framework recently (2022-11-07) announced that they will publish
a firmware upgrade to address some of the USB-C issues, including
power management. This could positively affect the above result,
improving both standby and runtime power usage.
The update came out in December 2022 and I redid my analysis with
the following results:
Device
Wattage
Amperage
Days
Note
baseline
0.25W
16mA
9
no cards, same as before upgrade
1 USB-C
0.25W
16mA
9
same as before
2 USB-C
0.25W
16mA
9
same
1 USB-A
0.80W
62mA
3
+550mW!! worse than before
2 USB-A
1.12W
73mA
<2
+320mW, on top of the above, bad!
Ethernet
0.62W
40mA
3-4
new result, decent
1TB SSD
0.52W
34mA
4
a bit worse than before (+2mA)
MicroSD
0.51W
22mA
4
same
DisplayPort
0.52W
34mA
4+
upgrade improved by 300mW
1 HDMI
?
38mA
?
same
2 HDMI
?
45mA
?
a bit worse than before (+3mA)
Normal
1.08W
70mA
~2
Ethernet, 2 USB-C, USB-A
Full results in standby-tests-306. The big takeaway for me is that
the update did not improve power usage on the USB-A ports which is a
big problem for my use case. There is a notable improvement on the
DisplayPort power consumption which brings it more in line with the
HDMI connector, but it still doesn't properly turn off on suspend
either.
Even worse, the USB-A ports now sometimes fails to resume after
suspend, which is pretty annoying. This is a known problem
that will hopefully get fixed in the final release.
I looked at building this myself but failed to run it. I opened a
RFP in Debian so that we can ship this in Debian, and also documented
my work there.
Note that there is now a counter that tracks charge/discharge
cycles. It's visible in tlp-stat -b, which is a nice
improvement:
Ethernet expansion card
The Framework ethernet expansion card is a fancy little doodle:
"2.5Gbit/s and 10/100/1000Mbit/s Ethernet", the "clear housing lets
you peek at the RTL8156 controller that powers it". Which is another
way to say "we didn't completely finish prod on this one, so it kind
of looks like we 3D-printed this in the shop"....
The card is a little bulky, but I guess that's inevitable considering
the RJ-45 form factor when compared to the thin Framework laptop.
I have had a serious issue when trying it at first: the link LEDs
just wouldn't come up. I made a full bug report in the forum and
with upstream support, but eventually figured it out on my own. It's
(of course) a power saving issue: if you reboot the machine, the links
come up when the laptop is running the BIOS POST check and even when
the Linux kernel boots.
I first thought that the problem is likely related to the
powertop service which I run at boot time to tweak some power saving
settings.
It seems like this:
By default, USB power saving is active in the kernel, but not
force-enabled for incompatible drivers. That is, devices that
support suspension will suspend, drivers that do not, will not.
So the fix is actually to uninstall tlp or disable that setting by
adding this to /etc/tlp.conf:
USB_AUTOSUSPEND=0
... but that disables auto-suspend on all USB devices, which may
hurt other power usage performance. I have found that a a
combination of:
USB_AUTOSUSPEND=1
USB_DENYLIST="0bda:8156"
and this on the kernel commandline:
usbcore.quirks=0bda:8156:k
... actually does work correctly. I now have this in my
/etc/default/grub.d/framework-tweaks.cfg file:
# net.ifnames=0: normal interface names ffs (e.g. eth0, wlan0, not wlp166
s0)
# nvme.noacpi=1: reduce SSD disk power usage (not working)
# mem_sleep_default=deep: reduce power usage during sleep (not working)
# usbcore.quirk is a workaround for the ethernet card suspend bug: https:
//guides.frame.work/Guide/Fedora+37+Installation+on+the+Framework+Laptop/
108?lang=en
GRUB_CMDLINE_LINUX="net.ifnames=0 nvme.noacpi=1 mem_sleep_default=deep usbcore.quirks=0bda:8156:k"
# fix the resolution in grub for fonts to not be tiny
GRUB_GFXMODE=1024x768
Other than that, I haven't been able to max out the card because I
don't have other 2.5Gbit/s equipment at home, which is strangely
satisfying. But running against my Turris Omnia
router, I could pretty much max a gigabit fairly easily:
The card doesn't require any proprietary firmware blobs which is
surprising. Other than the power saving issues, it just works.
In my power tests (see powerstat-wayland), the Ethernet card seems
to use about 1.6W of power idle, without link, in the above "quirky"
configuration where the card is functional but without autosuspend.
Proprietary firmware blobs
The framework does need proprietary firmware to operate. Specifically:
the WiFi network card shipped with the DIY kit is a AX210 card that
requires a 5.19 kernel or later, and the firmware-iwlwifi non-free firmware package
the Bluetooth adapter also loads the firmware-iwlwifi
package (untested)
the graphics work out of the box without firmware, but certain
power management features come only with special proprietary
firmware, normally shipped in the firmware-misc-nonfree
but currently missing from the package
Note that, at the time of writing, the latest i915 firmware from
linux-firmware has a serious bug where loading all the
accessible firmware results in noticeable I estimate 200-500ms lag
between the keyboard (not the mouse!) and the display. Symptoms also
include tearing and shearing of windows, it's pretty nasty.
One workaround is to delete the two affected firmware files:
cd /lib/firmware && rm adlp_guc_70.1.1.bin adlp_guc_69.0.3.bin
update-initramfs -u
You will get the following warning during build, which is good as
it means the problematic firmware is disabled:
W: Possible missing firmware /lib/firmware/i915/adlp_guc_69.0.3.bin for module i915
W: Possible missing firmware /lib/firmware/i915/adlp_guc_70.1.1.bin for module i915
But then it also means that critical firmware isn't loaded, which
means, among other things, a higher battery drain. I was able to move
from 8.5-10W down to the 7W range after making the firmware work
properly. This is also after turning the backlight all the way down,
as that takes a solid 2-3W in full blast.
The proper fix is to use some compositing manager. I ended up using
compton with the following systemd unit:
compton is orphaned however, so you might be tempted to use
picom instead, but in my experience the latter uses much
more power (1-2W extra, similar experience). I also tried
compiz but it would just crash with:
anarcat@angela:~$ compiz --replace
compiz (core) - Warn: No XI2 extension
compiz (core) - Error: Another composite manager is already running on screen: 0
compiz (core) - Fatal: No manageable screens found on display :0
When running from the base session, I would get this instead:
Also note that the iwlwifi firmware also looks incomplete. Even with
the package installed, I get those errors in dmesg:
[ 19.534429] Intel(R) Wireless WiFi driver for Linux
[ 19.534691] iwlwifi 0000:a6:00.0: enabling device (0000 -> 0002)
[ 19.541867] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-72.ucode (-2)
[ 19.541881] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-72.ucode (-2)
[ 19.541882] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-72.ucode failed with error -2
[ 19.541890] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-71.ucode (-2)
[ 19.541895] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-71.ucode (-2)
[ 19.541896] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-71.ucode failed with error -2
[ 19.541903] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-70.ucode (-2)
[ 19.541907] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-70.ucode (-2)
[ 19.541908] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-70.ucode failed with error -2
[ 19.541913] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-69.ucode (-2)
[ 19.541916] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-69.ucode (-2)
[ 19.541917] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-69.ucode failed with error -2
[ 19.541922] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-68.ucode (-2)
[ 19.541926] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-68.ucode (-2)
[ 19.541927] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-68.ucode failed with error -2
[ 19.541933] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-67.ucode (-2)
[ 19.541937] iwlwifi 0000:a6:00.0: firmware: failed to load iwlwifi-ty-a0-gf-a0-67.ucode (-2)
[ 19.541937] iwlwifi 0000:a6:00.0: Direct firmware load for iwlwifi-ty-a0-gf-a0-67.ucode failed with error -2
[ 19.544244] iwlwifi 0000:a6:00.0: firmware: direct-loading firmware iwlwifi-ty-a0-gf-a0-66.ucode
[ 19.544257] iwlwifi 0000:a6:00.0: api flags index 2 larger than supported by driver
[ 19.544270] iwlwifi 0000:a6:00.0: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.63.2.1
[ 19.544523] iwlwifi 0000:a6:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 19.544528] iwlwifi 0000:a6:00.0: firmware: failed to load iwl-debug-yoyo.bin (-2)
[ 19.544530] iwlwifi 0000:a6:00.0: loaded firmware version 66.55c64978.0 ty-a0-gf-a0-66.ucode op_mode iwlmvm
Some of those are available in the latest upstream firmware package
(iwlwifi-ty-a0-gf-a0-71.ucode, -68, and -67), but not all
(e.g. iwlwifi-ty-a0-gf-a0-72.ucode is missing) . It's unclear what
those do or don't, as the WiFi seems to work well without them.
I still copied them in from the latest linux-firmware package in the
hope they would help with power management, but I did not notice a
change after loading them.
There are also multiple knobs on the iwlwifi and iwlmvm
drivers. The latter has a power_schmeme setting which defaults to
2 (balanced), setting it to 3 (low power) could improve
battery usage as well, in theory. The iwlwifi driver also has
power_save (defaults to disabled) and power_level (1-5, defaults
to 1) settings. See also the output of modinfo iwlwifi and
modinfo iwlmvm for other driver options.
Graphics acceleration
After loading the latest upstream firmware and setting up a
compositing manager (compton, above), I tested the classic
glxgears.
Running in a window gives me odd results, as the gears basically grind
to a halt:
Running synchronized to the vertical refresh. The framerate should be
approximately the same as the monitor refresh rate.
137 frames in 5.1 seconds = 26.984 FPS
27 frames in 5.4 seconds = 5.022 FPS
Ouch. 5FPS!
But interestingly, once the window is in full screen, it does hit the
monitor refresh rate:
300 frames in 5.0 seconds = 60.000 FPS
I'm not really a gamer and I'm not normally using any of that fancy
graphics acceleration stuff (except maybe my browser does?).
I installed intel-gpu-tools for the intel_gpu_top
command to confirm the GPU was engaged when doing those simulations. A
nice find. Other useful diagnostic tools include glxgears and
glxinfo (in mesa-utils) and (vainfo in vainfo).
Following to this post, I also made sure to have those settings
in my about:config in Firefox, or, in user.js:
user_pref("media.ffmpeg.vaapi.enabled", true);
Note that the guide suggests many other settings to tweak, but those
might actually be overkill, see this comment and its parents. I
did try forcing hardware acceleration by setting gfx.webrender.all
to true, but everything became choppy and weird.
The guide also mentions installing the intel-media-driver package,
but I could not find that in Debian.
The Arch wiki has, as usual, an excellent reference on hardware
acceleration in Firefox.
Chromium / Signal desktop bugs
It looks like both Chromium and Signal Desktop misbehave with my
compositor setup (compton + i3). The fix is to add a persistent
flag to Chromium. In Arch, it's conveniently in
~/.config/chromium-flags.conf but that doesn't actually work in
Debian. I had to put the flag in
/etc/chromium.d/disable-compositing, like this:
It's possible another one of the hundreds of flags might fix this
issue better, but I don't really have time to go through this entire,
incomplete, and unofficial list (!?!).
Signal Desktop is a similar problem, and doesn't reuse those flags
(because of course it doesn't). Instead I had to rewrite the wrapper
script in /usr/local/bin/signal-desktop to use this instead:
exec /usr/bin/flatpak run --branch=stable --arch=x86_64 org.signal.Signal --disable-gpu-compositing "$@"
This was mostly done in this Puppet commit.
I haven't figured out the root of this problem. I did try using
picom and xcompmgr; they both suffer from the same issue. Another
Debian testing user on Wayland told me they haven't seen this problem,
so hopefully this can be fixed by switching to
wayland.
Graphics card hangs
I believe I might have this bug which results in a total
graphical hang for 15-30 seconds. It's fairly rare so it's not too
disruptive, but when it does happen, it's pretty alarming.
The comments on that bug report are encouraging though: it seems this
is a bug in either mesa or the Intel graphics driver, which means many
people have this problem so it's likely to be fixed. There's actually
a merge request on mesa already (2022-12-29).
It could also be that bug because the error message I get is
actually:
Jan 20 12:49:10 angela kernel: Asynchronous wait on fence 0000:00:02.0:sway[104431]:cb0ae timed out (hint:intel_atomic_commit_ready [i915])
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] GPU HANG: ecode 12:0:00000000
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] Resetting chip for stopped heartbeat on rcs0
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] GuC firmware i915/adlp_guc_70.1.1.bin version 70.1
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] HuC firmware i915/tgl_huc_7.9.3.bin version 7.9
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] HuC authenticated
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] GuC submission enabled
Jan 20 12:49:15 angela kernel: i915 0000:00:02.0: [drm] GuC SLPC enabled
It's a solid 30 seconds graphical hang. Maybe the keyboard and
everything else keeps working. The latter bug report is quite long,
with many comments, but this one from January 2023 seems to say
that Sway 1.8 fixed the problem. There's also an earlier patch to
add an extra kernel parameter that supposedly fixes that too. There's
all sorts of other workarounds in there, for example this:
from this comment... So that one is unsolved, as far as the
upstream drivers are concerned, but maybe could be fixed through Sway.
Weird USB hangs / graphical glitches
I have had weird connectivity glitches better described in this
post, but basically: my USB keyboard and mice (connected over a
USB hub) drop keys, lag a lot or hang, and I get visual glitches.
The fix was to tighten the screws around the CPU on the motherboard
(!), which is, thankfully, a rather simple repair.
USB docks are hell
Note that the monitors are hooked up to angela through a USB-C /
Thunderbolt dock from Cable Matters, with the lovely name of
201053-SIL. It has issues, see this blog
post for an in-depth discussion.
Shipping details
I ordered the Framework in August 2022 and received it about a month
later, which is sooner than expected because the August batch was
late.
People (including me) expected this to have an impact on the September
batch, but it seems Framework have been able to fix the delivery
problems and keep up with the demand.
As of early 2023, their website announces that laptops ship "within 5
days". I have myself ordered a few expansion cards in November 2022,
and they shipped on the same day, arriving 3-4 days later.
The supply pipeline
There are basically 6 steps in the Framework shipping pipeline, each
(except the last) accompanied with an email notification:
pre-order
preparing batch
preparing order
payment complete
shipping
(received)
This comes from the crowdsourced spreadsheet, which should be
updated when the status changes here.
I was part of the "third batch" of the 12th generation laptop, which
was supposed to ship in September. It ended up arriving on my door
step on September 27th, about 33 days after ordering.
It seems current orders are not processed in "batches", but in real
time, see this blog post for details on shipping.
Shipping trivia
I don't know about the others, but my laptop shipped through no less
than four different airplane flights. Here are the hops it took:
I can't quite figure out how to calculate exactly how much mileage
that is, but it's huge. The ride through Alaska is surprising enough
but the bounce back through Winnipeg is especially weird. I guess
the route happens that way because of Fedex shipping hubs.
There was a related oddity when I had my Purism laptop shipped: it
left from the west coast and seemed to enter on an endless, two week
long road trip across the continental US.
Interested in future updates? Follow me on mastodon at
@paul@soylent.green. Posts about
hz.tools will be tagged
#hztools.
If you're on the Fediverse, I'd very much appreciate boosts on
my announcement toot!
Ever since 2019, I ve been learning about how radios work, and trying to learn
about using them the hard way by writing as much of the stack as is
practical (for some value of practical) myself. I wrote my first Hello World
in 2018, which was a simple FM radio player, which used librtlsdr to read in
an IQ stream, did some filtering, and played the real valued audio stream via
pulseaudio. Over 4 years this has slowly grown through persistence, lots of
questions to too many friends to thank (although I will try), and the eternal
patience of my wife hearing about radios nonstop for years into a number
of Go repos that can do quite a bit, and support a handful of radios.
I ve resisted making the repos public not out of embarrassment or a desire to
keep secrets, but rather, an attempt to keep myself free of any maintenance
obligations to users so that I could freely break my own API, add and remove
API surface as I saw fit. The worst case was to have this project feel like
work, and I can t imagine that will happen if I feel frustrated by PRs
that are getting ahead of me solving problems I didn t yet know about, or
bugs I didn t understand the fix for.
As my rate of changes to the most central dependencies has slowed, i ve begun
to entertain the idea of publishing them. After a bit of back and forth, I ve
decided
it s time to make a number of them public,
and to start working on them in the open, as I ve built up a bit of knowledge
in the space, and I and feel confident that the repo doesn t contain overt
lies. That s not to say it doesn t contain lies, but those lies are likely
hidden and lurking in the dark. Beware.
That being said, it shouldn t be a surprise to say I ve not published
everything yet for the same reasons as above. I plan to open repos as the
rate of changes slows and I understand the problems the library solves well
enough or if the project dead ends and I ve stopped learning.
Intention behind hz.tools
It s my sincere hope that my repos help to make Software Defined Radio (SDR)
code a bit easier to understand, and serves as an understandable framework to
learn with. It s a large codebase, but one that is possible to sit down and
understand because, well, it was written by a single person. Frankly, I m also
not productive enough in my free time in the middle of the night and on
weekends and holidays to create a codebase that s too large to understand, I
hope!
I remain wary of this project turning into work, so my goal is to be very
upfront about my boundaries, and the limits of what classes of contributions
i m interested in seeing.
Here s some goals of open sourcing these repos:
I do want this library to be used to learn with. Please go through it
all and use it to learn about radios and how software can control them!
I am interested in bugs if there s a problem you discover. Such bugs
are likely a great chance for me to fix something I ve misunderstood or
typoed.
I am interested in PRs fixing bugs you find. I may need a bit of a
back and forth to fully understand the problem if I do not understand
the bug and fix yet. I hope you may have some grace if it s taking a long
time.
Here s a list of some anti-goals of open sourcing these repos.
I do not want this library to become a critical dependency of an
important project, since I do not have the time to deal with the maintenance
burden. Putting me in that position is going to make me very uncomfortable.
I am not interested in feature requests, the features have grown as
I ve hit problems, I m not interested in building or maintaining features
for features sake. The API surface should be exposed enough to allow others
to experiment with such things out-of-tree.
I m not interested in clever code replacing clear code without a very
compelling reason.
I use GNU/Linux (specifically Debian
),
and from time-to-time I ve made sure that my code runs on OpenBSD
too.
Platforms beyond that will likely not be supported at the expense of either
of those two. I ll take fixes for bugs that fix a problem on another
platform, but not damage the code to work around issues / lack of features
on other platforms (like Windows).
I m not saying all this to be a jerk, I do it to
make sure I can continue on my journey to learn about how radios work without
my full time job becoming maintaining a radio framework single-handedly for
other people to use even if it means I need to close PRs or bugs without
merging it or fixing the issue.
With all that out of the way, I m very happy to announce that the repos are now
public under github.com/hztools.
Should you use this?
Probably not. The intent here is not to provide a general purpose Go SDR
framework for everyone to build on, although I am keenly aware it looks and
feels like it, since that what it is to me. This is a learning project, so for
any use beyond joining me in learning should use something like
GNU Radio
or a similar framework that has a community behind it.
In fact, I suspect most contributors ought to be contributing to GNU Radio, and
not this project. If I can encourage people to do so,
contribute to GNU Radio!
Nothing makes me happier than seeing GNU Radio continue to be the go-to, and
well supported. Consider
donating to GNU Radio!
hz.tools/rf - Frequency types
The hz.tools/rf library contains the abstract concept of frequency, and
some very basic helpers to interact with frequency ranges (such as helpers
to deal with frequency ranges, or frequency range math) as well as frequencies
and some very basic conversions (to meters, etc) and parsers (to parse values
like 10MHz). This ensures that all the hz.tools libraries have a shared
understanding of Frequencies, a standard way of representing ranges of
Frequencies, and the ability to handle the IO boundary with things like CLI
arguments, JSON or YAML.
The git repo can be found at
github.com/hztools/go-rf, and is
importable as hz.tools/rf.
// Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
freq:=rf.MustParseHz("-10kHz")
fmt.Printf("Frequency: %s\n", freq+rf.MHz)
// Prints: 'Frequency: 990kHz'
// Return the Intersection between two RF ranges, and print
// it to stdout.
r1:=rf.Rangerf.KHz, rf.MHzr2:=rf.Rangerf.Hz(10), rf.KHz*100fmt.Printf("Range: %s\n", r1.Intersection(r2))
// Prints: Range: 1000Hz->100kHz
These can be used to represent tons of things - ranges can be used for things
like the tunable range of an SDR, the bandpass of a filter or the frequencies
that correspond to a bin of an FFT, while frequencies can be used for things
such as frequency offsets or the tuned center frequency.
hz.tools/sdr - SDR I/O and IQ Types
This is the big one. This library represents the majority of the shared
types and bindings, and is likely the most useful place to look at when
learning about the IO boundary between a program and an SDR.
The git repo can be found at
github.com/hztools/go-sdr, and is
importable as hz.tools/sdr.
This library is designed to look (and in some cases, mirror) the Go io idioms
so that this library feels as idiomatic as it can, so that Go builtins interact
with IQ in a way that s possible to reason about, and to avoid reinventing the
wheel by designing new API surface. While some of the API looks (and is even
called) the same thing as a similar function in io, the implementation is
usually a lot more naive, and may have unexpected sharp edges such as
concurrency issues or performance problems.
The following IQ types are implemented using the sdr.Samples interface. The
hz.tools/sdr package contains helpers for conversion between types, and some
basic manipulation of IQ streams.
IQ Format
hz.tools Name
Underlying Go Type
Interleaved uint8 (rtl-sdr)
sdr.SamplesU8
[][2]uint8
Interleaved int8 (hackrf, uhd)
sdr.SamplesI8
[][2]int8
Interleaved int16 (pluto, uhd)
sdr.SamplesI16
[][2]int16
Interleaved float32 (airspy, uhd)
sdr.SamplesC64
[]complex64
The following SDRs have implemented drivers in-tree.
SDR
Format
RX/TX
State
rtl
u8
RX
Good
HackRF
i8
RX/TX
Good
PlutoSDR
i16
RX/TX
Good
rtl kerberos
u8
RX
Old
uhd
i16/c64/i8
RX/TX
Good
airspyhf
c64
RX
Exp
The following major packages and subpackages exist at the time of writing:
Import
What is it?
hz.tools/sdr
Core IQ types, supporting types and implementations that interact with the byte boundary
hz.tools/sdr/rtl
sdr.Receiver implementation using librtlsdr.
hz.tools/sdr/rtl/kerberos
Helpers to enable coherent RX using the Kerberos SDR.
hz.tools/sdr/rtl/e4k
Helpers to interact with the E4000 RTL-SDR dongle.
hz.tools/sdr/fft
Interfaces for performing an FFT, which are implemented by other packages.
sdr.Transceiver implementation for the PlutoSDR using libiio.
hz.tools/sdr/uhd
sdr.Transceiver implementation for UHD radios, specifically the B210 and B200mini
hz.tools/sdr/hackrf
sdr.Transceiver implementation for the HackRF using libhackrf.
hz.tools/sdr/mock
Mock SDR for testing purposes.
hz.tools/sdr/airspyhf
sdr.Receiver implementation for the AirspyHF+ Discovery with libairspyhf.
hz.tools/sdr/internal/simd
SIMD helpers for IQ operations, written in Go ASM. This isn t the best to learn from, and it contains pure go implemtnations alongside.
hz.tools/sdr/stream
Common Reader/Writer helpers that operate on IQ streams.
hz.tools/fftw - hz.tools/sdr/fft implementation
The hz.tools/fftw package contains bindings to libfftw3 to implement
the hz.tools/sdr/fft.Planner type to transform between the time and
frequency domain.
The git repo can be found at
github.com/hztools/go-fftw, and is
importable as hz.tools/fftw.
This is the default throughout most of my codebase, although that default is
only expressed at the leaf package libraries should not be hardcoding the
use of this library in favor of taking an fft.Planner, unless it s used as
part of testing. There are a bunch of ways to do an FFT out there, things like
clFFT or a pure-go FFT implementation could be plugged in depending on what s
being solved for.
hz.tools/ fm,am - analog audio demodulation and modulation
The hz.tools/fm and hz.tools/am packages contain demodulators for
AM analog radio, and FM analog radio. This code is a bit old, so it has
a lot of room for cleanup, but it ll do a very basic demodulation of IQ
to audio.
The git repos can be found at
github.com/hztools/go-fm and
github.com/hztools/go-am,
and are importable as
hz.tools/fm and
hz.tools/am.
As a bonus, the hz.tools/fm package also contains a modulator, which has been
tested on the air and with some of my handheld radios. This code is a bit
old, since the hz.tools/fm code is effectively the first IQ processing code
I d ever written, but it still runs and I run it from time to time.
// Basic sketch for playing FM radio using a reader stream from
// an SDR or other IQ stream.
bandwidth:=150*rf.KHzreader, err = stream.ConvertReader(reader, sdr.SampleFormatC64)
iferr!=nil...demod, err:=fm.Demodulate(reader, fm.DemodulatorConfigDeviation: bandwidth/2,
Downsample: 8, // some value here depending on sample rate
Planner: fftw.Plan,
)
iferr!=nil...speaker, err:=pulseaudio.NewWriter(pulseaudio.ConfigFormat: pulseaudio.SampleFormatFloat32NE,
Rate: demod.SampleRate(),
AppName: "rf",
StreamName: "fm",
Channels: 1,
SinkName: "",
)
iferr!=nil...buf:= make([]float32, 1024*64)
fori, err:=demod.Read(buf)
iferr!=nil...ifi==0
panic("...")
iferr:=speaker.Write(buf[:i]); err!=nil...
hz.tools/rfcap - byte serialization for IQ data
The hz.tools/rfcap package is the reference implementation of the
rfcap spec , and is how I store IQ captures
locally, and how I send them across a byte boundary.
The git repo can be found at
github.com/hztools/go-rfcap, and is
importable as hz.tools/rfcap.
If you re interested in storing IQ in a way others can use, the better approach
is to use SigMFrfcap exists for cases
like using UNIX pipes to move IQ around, through APIs, or when I send
IQ data through an OS socket, to ensure the sample format (and other metadata)
is communicated with it.
rfcap has a number of limitations, for instance, it can not express a change
in frequency or sample rate during the capture, since the header is fixed at
the beginning of the file.
I m happy to announce that Netplan version 0.106 is now available on GitHub and is soon to be deployed into an Ubuntu/Debian/Fedora installation near you! Six months and 65 commits after the previous version, this release is brought to you by 4 free software contributors from around the globe.
Highlights
Highlights of this release include the new netplan status command, which queries your system for IP addresses, routes, DNS information, etc in addition to the Netplan backend renderer (NetworkManager/networkd) in use and the relevant Netplan YAML configuration ID. It displays all this in a nicely formatted way (or alternatively in machine readable YAML/JSON format).
Furthermore, we implemented a clean libnetplan API which can be used by external tools to parse Netplan configuration, migrated away from non-inclusive language (PR#303) and improved the overall Netplan documentation. Another change that should be noted, is that the match.macaddress stanza now only matches on PermanentMACAddress=on the systemd-networkd backend, as has been the case on the NetworkManager backend ever since (see PR#278 for background information on this slight change in behavior).
Changelog
Akvorado collects network flows using IPFIX or sFlow. It stores them
in a ClickHouse database. A web console allows a user to query the data and
plot some graphs. A nice aspect of this console is how we can filter flows with
a SQL-like language:
Filter editor in Akvorado console
Often, web interfaces expose a querybuilder to build such filters. I think combining a
SQL-like language with an editor supporting completion, syntax
highlighting, and linting is a better approach.1
The language parser is built with pigeon (Go) from a parsing expression
grammar or PEG. The editor component is CodeMirror (TypeScript).
Language parser
PEG grammars are relatively recent2 and are an alternative to
context-free grammars. They are easier to write and they can generate better
error messages. Python switched from an LL(1)-based parser to a PEG-based
parser in Python 3.9.
pigeon generates a parser for Go. A grammar is a set of rules. Each rule is
an identifier, with an optional user-friendly label for error messages, an
expression, and an action in Go to be executed on match. You can find the
complete grammar in parser.peg. Here is a simplified rule:
The rule identifier is ConditionIPExpr. It case-insensitively matches
ExporterAddress, SrcAddr, or DstAddr. The action for each case returns the
proper case for the column name. That s what is stored in the column variable.
Then, it matches one of the possible operators. As there is no code block, it
stores the matched string directly in the operator variable. Then, it tries to
match the IP rule, which is defined elsewhere in the grammar. If it succeeds,
it stores the result of the match in the ip variable and executes the final
action. The action turns the column, operator, and IP into a proper expression
for ClickHouse. For example, if we have ExporterAddress = 203.0.113.15, we
get ExporterAddress = IPv6StringToNum('203.0.113.15').
The IP rule uses a rudimentary regular expression but checks if the matched
address is correct in the action block, thanks to netip.ParseAddr():
IP "IP address"[0-9A-Fa-f:.]+ ip,err:=netip.ParseAddr(string(c.text))iferr!=nilreturn"",errors.New("expecting an IP address")returnip.String(),nil
Our parser safely turns the filter into a WHERE clause accepted by
ClickHouse:3
Integration in CodeMirrorCodeMirror is a versatile code editor that can be easily integrated into
JavaScript projects. In Akvorado, the Vue.js component,
InputFilter, uses CodeMirror as its foundation and
leverages features such as syntax highlighting, linting, and completion. The
source code for these capabilities can be found in the
codemirror/lang-filter/ directory.
Syntax highlighting
The PEG grammar for Go cannot be utilized directly4 and the requirements
for parsers for editors are distinct: they should be error-tolerant and operate
incrementally, as code is typically updated character by character. CodeMirror
offers a solution through its own parser generator, Lezer.
We don t need this additional parser to fully understand the filter language.
Only the basic structure is needed: column names, comparison and logic
operators, quoted and unquoted values. The grammar is therefore quite short
and does not need to be updated often:
@topFilter
expression
expression
Not expression
"(" expression ")""(" expression ")" And expression
"(" expression ")" Or expression
comparisonExpression And expression
comparisonExpression Or expression
comparisonExpression
comparisonExpression
Column Operator Value
Value
String Literal ValueLParen ListOfValues ValueRParen
ListOfValues
ListOfValues ValueComma (String Literal)
String Literal
// [ ]@tokens// [ ]Column std.asciiLetter (std.asciiLetter std.digit)* Operator $[a-zA-Z!=><]+ String'"'(![\\\n"]"\\" _)* '"'?
"'"(![\\\n']"\\" _)* "'"?
Literal(std.digit std.asciiLetter $[.:/])+ // [ ]
The expression SrcAS = 12322 AND (DstAS = 1299 OR SrcAS = 29447) is parsed to:
Linting
We offload linting to the original parser in Go. The
/api/v0/console/filter/validate endpoint accepts a filter and returns a JSON
structure with the errors that were found:
"message":"at line 1, position 12: string literal not terminated","errors":[ "line":1,"column":12,"offset":11,"message":"string literal not terminated", ]
Completion
The completion system takes a hybrid approach. It splits the work between the
frontend and the backend to offer useful suggestions for completing filters.
The frontend uses the parser built with Lezer to determine the context of
the completion: do we complete a column name, an operator, or a value? It also
extracts the column name if we are completing something else. It forwards the
result to the backend through the /api/v0/console/filter/complete endpoint.
Walking the syntax tree was not as easy as I thought, but unit tests helped
a lot.
The backend uses the parser generated by pigeon to complete a column name
or a comparison operator. For values, the completions are either static or
extracted from the ClickHouse database. A user can complete an AS number from
an organization name thanks to the following snippet:
results:=[]structLabelstring ch:"label" Detailstring ch:"detail" columnName:="DstAS"sqlQuery:=fmt.Sprintf( SELECT concat('AS', toString(%s)) AS label, dictGet('asns', 'name', %s) AS detail FROM flows WHERE TimeReceived > date_sub(minute, 1, now()) AND detail != '' AND positionCaseInsensitive(detail, $1) >= 1 GROUP BY label, detail ORDER BY COUNT(*) DESC LIMIT 20,columnName,columnName)iferr:=conn.Select(ctx,&results,sqlQuery,input.Prefix);err!=nilc.r.Err(err).Msg("unable to query database")breakfor_,result:=rangeresultscompletions=append(completions,filterCompletionLabel:result.Label,Detail:result.Detail,Quoted:false, )
In my opinion, the completion system is a major factor in making the field
editor an efficient way to select flows. While a query builder may have been
more beginner-friendly, the completion system s ease of use and functionality
make it more enjoyable to use once you become familiar.
Moreover, building a query builder did not seem like a fun task for me.
This is the second part of how I build a read-only root setup for my router. You might want to read part 1 first, which covers the initial boot and general overview of how I tie the pieces together. This post will describe how I build the squashfs image that forms the main filesystem.
Most of the build is driven from a script, make-router, which I ll dissect below. It s highly tailored to my needs, and this is a fairly lengthy post, but hopefully the steps I describe prove useful to anyone trying to do something similar.
Breakdown of make-router
#!/bin/bash# Either rb3011 (arm) or rb5009 (arm64)#HOSTNAME="rb3011"HOSTNAME="rb5009"if["x$ HOSTNAME"=="xrb3011"];then
ARCH=armhf
elif["x$ HOSTNAME"=="xrb5009"];then
ARCH=arm64
else
echo"Unknown host: $ HOSTNAME"exit 1
fi
It s a bash script, and I allow building for either my RB3011 or RB5009, which means a different architecture (32 vs 64 bit). I run this script on my Pi 4 which means I don t have to mess about with QemuUserEmulation.
BASE_DIR=$(dirname$0)IMAGE_FILE=$(mktemp--tmpdir router.$ ARCH.XXXXXXXXXX.img)MOUNT_POINT=$(mktemp-p /mnt -d router.$ ARCH.XXXXXXXXXX)# Build and mount an ext4 image file to put the root file system indd if=/dev/zero bs=1 count=0 seek=1G of=$ IMAGE_FILE
mkfs -t ext4 $ IMAGE_FILE
mount -o loop $ IMAGE_FILE$ MOUNT_POINT
I build the image in a loopback ext4 file on tmpfs (my Pi4 is the 8G model), which makes things a bit faster.
# Add dpkg excludesmkdir-p$ MOUNT_POINT/etc/dpkg/dpkg.cfg.d/
cat<<EOF > $ MOUNT_POINT/etc/dpkg/dpkg.cfg.d/path-excludes
# Exclude docs
path-exclude=/usr/share/doc/*
# Only locale we want is English
path-exclude=/usr/share/locale/*
path-include=/usr/share/locale/en*/*
path-include=/usr/share/locale/locale.alias
# No man pages
path-exclude=/usr/share/man/*
EOF
Create a dpkg excludes config to drop docs, man pages and most locales before we even start the bootstrap.
Actually do the debootstrap step, including a bunch of extra packages that we want.
# Install mqtt-arpcp$ BASE_DIR/debs/mqtt-arp_1_$ ARCH.deb $ MOUNT_POINT/tmp
chroot$ MOUNT_POINT dpkg -i /tmp/mqtt-arp_1_$ ARCH.deb
rm$ MOUNT_POINT/tmp/mqtt-arp_1_$ ARCH.deb
# Frob the mqtt-arp config so it starts after mosquittosed-i-e's/After=.*/After=mosquitto.service/'$ MOUNT_POINT/lib/systemd/system/mqtt-arp.service
I haven t uploaded mqtt-arp to Debian, so I install a locally built package, and ensure it starts after mosquitto (the MQTT broker), given they re running on the same host.
# Frob watchdog so it starts earlier than multi-usersed-i-e's/After=.*/After=basic.target/'$ MOUNT_POINT/lib/systemd/system/watchdog.service
# Make sure the watchdog is poking the device filesed-i-e's/^#watchdog-device/watchdog-device/'$ MOUNT_POINT/etc/watchdog.conf
watchdog timeouts were particularly an issue on the RB3011, where the default timeout didn t give enough time to reach multiuser mode before it would reset the router. Not helpful, so alter the config to start it earlier (and make sure it s configured to actually kick the device file).
# Clean up docs + localesrm-r$ MOUNT_POINT/usr/share/doc/*rm-r$ MOUNT_POINT/usr/share/man/*for dir in$ MOUNT_POINT/usr/share/locale/*/;do
if["$ dir"!="$ MOUNT_POINT/usr/share/locale/en/"];then
rm-r$ dirfi
done
Clean up any docs etc that ended up installed.
# Set root password to rootecho"root:root"chroot$ MOUNT_POINT chpasswd
The only login method is ssh key to the root account though I suppose this allows for someone to execute a privilege escalation from a daemon user so I should probably randomise this. Does need to be known though so it s possible to login via the serial console for debugging.
There are config files that are easier to replace wholesale, some of which are specific to the hardware (e.g. related to network interfaces). See below for some more details.
# Build symlinks into flash for boot / modulesln-s /mnt/flash/lib/modules $ MOUNT_POINT/lib/modules
rmdir$ MOUNT_POINT/boot
ln-s /mnt/flash/boot $ MOUNT_POINT/boot
The kernel + its modules live outside the squashfs image, on the USB flash drive that the image lives on. That makes for easier kernel upgrades.
# Put our git revision into os-releaseecho-n"GIT_VERSION=">>$ MOUNT_POINT/etc/os-release
(cd$ BASE_DIR; git describe --tags)>>$ MOUNT_POINT/etc/os-release
Always helpful to be able to check the image itself for what it was built from.
# Add some stuff to root's .bashrccat<<EOF >> $ MOUNT_POINT/root/.bashrc
alias ls='ls -F --color=auto'
eval "\$(dircolors)"
case "\$TERM" in
xterm* rxvt*)
PS1="\\[\\e]0;\\u@\\h: \\w\a\\]\$PS1"
;;
*)
;;
esac
EOF
Just some niceties for when I do end up logging in.
# Save the installed package list offchroot$ MOUNT_POINT dpkg --get-selections> /tmp/wip-installed-packages
Save off the installed package list. This was particularly useful when trying to replicate the existing router setup and making sure I had all the important packages installed. It doesn t really serve a purpose now.
In terms of the config files I copy into /etc, shared across both routers are the following:
Breakdown of shared config
FTP master
This month I accepted 419 and rejected 46 packages. The overall number of packages that got accepted was 429. Looking at these numbers and comparing them to the previous month, one can see: the freeze is near. Everybody wants to get some packages into the archive and I hope nobody is disappointed.
Debian LTS
This was my hundred-third month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.
This month my all in all workload has been 14h.
During that time I uploaded:
[DLA 3272-1] sudo (embargoed) security update for one CVE
[DLA 3286-1] tor security update for one CVE
[DLA 3290-1] libzen security update for one CVE
[libzen Bullseye] debdiff sent to maintainer
[DLA 3294-1] libarchive security update for one CVE
I also attended the monthly LTS meeting and did some days of frontdesk duties.
Debian ELTS
This month was the fifty fourth ELTS month.
[ELA-772-1] sudo security update of Jessie and Stretch for one CVE
[ELA-781-1] libzen security update of Stretch for one CVE
[ELA-782-1] xorg-server security update of Jessie and Stretch for six CVEs
[ELA-790-1] libarchive security update of Jessie and Stretch for one CVEs
Last but not least I did some days of frontdesk duties.
Debian Astro
This month I uploaded improved packages or new versions of:
These new packages are already preparations for the new CUPS architecture. This architecture will be introduced after the release of Bookworm and shall be ready at least in Trixie. Parts of this work is generously funded by Freexian!
Other stuff
This month I uploaded improved packages of:
Welcome to the first report for 2023 from the Reproducible Builds project!
In these reports we try and outline the most important things that we have been up to over the past month, as well as the most important things in/around the community. As a quick recap, the motivation behind the reproducible builds effort is to ensure no malicious flaws can be deliberately introduced during compilation and distribution of the software that we run on our devices. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.
News
In a curious turn of events, GitHub first announced this month that the checksums of various Git archives may be subject to change, specifically that because:
the default compression for Git archives has recently changed. As result, archives downloaded from GitHub may have different checksums even though the contents are completely unchanged.
This change (which was brought up on our mailing list last October) would have had quite wide-ranging implications for anyone wishing to validate and verify downloaded archives using cryptographic signatures. However, GitHub reversed this decision, updating their original announcement with a message that We are reverting this change for now. More details to follow. It appears that this was informed in part by an in-depth discussion in the GitHub Community issue tracker.
The Bundesamt f r Sicherheit in der Informationstechnik (BSI) (trans: The Federal Office for Information Security ) is the agency in charge of managing computer and communication security for the German federal government. They recently produced a report that touches on attacks on software supply-chains (Supply-Chain-Angriff). (German PDF)
Contributor Seb35 updated our website to fix broken links to Tails Git repository [][], and Holger updated a large number of pages around our recent summit in Venice [][][][].
Noak J nsson has written an interesting paper entitled The State of Software Diversity in the Software Supply Chain of Ethereum Clients. As the paper outlines:
In this report, the software supply chains of the most popular Ethereum clients are cataloged and analyzed. The dependency graphs of Ethereum clients developed in Go, Rust, and Java, are studied. These client are Geth, Prysm, OpenEthereum, Lighthouse, Besu, and Teku. To do so, their dependency graphs are transformed into a unified format. Quantitative metrics are used to depict the software supply chain of the blockchain. The results show a clear difference in the size of the software supply chain required for the execution layer and consensus layer of Ethereum.
F-Droid & Android
There was a very large number of changes in the F-Droid and wider Android ecosystem this month:
On January 15th, a blog post entitled Towards a reproducible F-Droid was published on the F-Droid website, outlining the reasons why F-Droid signs published APKs with its own keys and how reproducible builds allow using upstream developers keys instead. In particular:
GitLab user Parwor discovered that the number of CPU cores can affect the reproducibility of .dex files. []
FC Stegerman also announced the 0.2.0 and 0.2.1 releases of reproducible-apk-tools, a suite of tools to help make .apk files reproducible. Several new subcommands and scripts were added, and a number of bugs were fixed as well [][]. They also updated the F-Droid website to improve the reproducibility-related documentation. [][]
A number of bugs related to reproducibility were discovered in Android itself. Firstly, the non-deterministic order of .zip entries in .apk files [] and then newline differences between building on Windows versus Linux that can make builds not reproducible as well. [] (Note that these links may require a Google account to view.)
Debian
As mentioned in last month s report, Vagrant Cascadian has been organising a series of online sprints in order to clear the huge backlog of reproducible builds patches submitted by performing NMUs (Non-Maintainer Uploads). During January, a sprint took place on the 10th, resulting in the following uploads:
During this sprint, Holger Levsen filed Debian bug #1028615 to request that the tracker.debian.org service display results of reproducible rebuilds, not just reproducible CI results.
Elsewhere in Debian, strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, version 1.13.1-1 was uploaded to Debian unstable by Holger Levsen, including a fix by FC Stegerman (obfusk) to update a regular expression for the latest version of file(1) []. (#1028892)
Lastly, 65 reviews of Debian packages were added, 21 were updated and 35 were removed this month adding to our knowledge about identified issues.
Finally, an existing tool called rpmreproduce was (re-)discovered this month, which claims that given a buildinfo file from a RPM package, [it can] generate instructions for attempting to reproduce the binary packages built from the associated source and build information.
diffoscopediffoscope is our in-depth and content-aware diff utility. Not only can it locate and diagnose reproducibility issues, it can provide human-readable diffs from many kinds of binary formats. This month, Chris Lamb made the following changes to diffoscope, including preparing and uploading versions 231, 232, 233 and 234 to Debian:
No need for from __future__ import print_function import anymore. []
Comment and tidy the extras_require.json handling. []
Split inline Python code to generate test Recommends into a separate Python script. []
Update debian/tests/control after merging support for PyPDF support. []
Correctly catch segfaulting cd-iccdump binary. []
Drop some old debugging code. []
Allow ICC tests to (temporarily) fail. []
In addition, FC Stegerman (obfusk) made a number of changes, including:
Updating the test_text_proper_indentation test to support the latest version(s) of file(1). []
Use an extras_require.json file to store some build/release metadata, instead of accessing the internet. []
Updating an APK-related file(1) regular expression. []
Lastly, Sam James added support for PyPDF version 3 [] and Vagrant Cascadian updated a handful of tool references for GNU Guix. [][]
Upstream patches
The Reproducible Builds project attempts to fix as many currently-unreproducible packages as possible. This month, we wrote a large number of such patches, including:
Several patches for file(1) (which is used by reproducible builds tools like diffoscope and strip-nondeterminism) that improve detection of various file formats are now included in the Debian packaging. []
Testing framework
The Reproducible Builds project operates a comprehensive testing framework at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In January, the following changes were made by Holger Levsen:
Node changes:
Add three new nodes hosted at the Oregon State University Open Source Lab including integrating them into the DNS, maintenance and monitoring systems. [][][][][]
Debian-related changes:
Only keep diffoscope s HTML output (ie. no .json or .txt) for LTS suites and older in order to save diskspace on the Jenkins host. []
Re-create pbuilder base less frequently for the stretch, bookworm and experimental suites. []
OpenWrt-related changes:
Add gcc-multilib to OPENWRT_HOST_PACKAGES and install it on the nodes that need it. []
Detect more problems in the health check when failing to build OpenWrt. []
Misc changes:
Update the chroot-run script to correctly manage /dev and /dev/pts. [][][]
Update the Jenkins shell monitor script to collect disk stats less frequently [] and to include various directory stats. [][]
Update the real year in the configuration in order to be able to detect whether a node is running in the future or not. []
Bump copyright years in the default page footer. []
In addition, Christian Marangi submitted a patch to build OpenWrt packages with the V=s flag to enable debugging. []
If you are interested in contributing to the Reproducible Builds project, please visit the Contribute page on our website. You can get in touch with us via:
A new release 0.2.18 of RInside
arrived on CRAN and in Debian today. This is the first
release in ten months since the 0.2.17 release. RInside
provides a set of convenience classes which facilitate embedding of R inside of C++ applications and
programs, using the classes and functions provided by Rcpp.
This release brings a contributed change to how the internal REPL is
called: Dominick found the
current form more reliable when embedding R on Windows. We also updated
a few other things around the package.
The list of changes since the last release:
Changes in RInside
version 0.2.18 (2023-02-01)
The random number initialization was updated as in R.
The main REPL is now running via 'run_Rmainloop()'.
Small routine update to package and continuous
integration.
I ve always found the operation of apt software package repositories to be a mystery. There appears to be a lack of transparency into which people have access to important apt package repositories out there, how the automatic non-human update mechanism is implemented, and what changes are published. I m thinking of big distributions like Ubuntu and Debian, but also the free GNU/Linux distributions like Trisquel and PureOS that are derived from the more well-known distributions.
As far as I can tell, anyone who has the OpenPGP private key trusted by a apt-based GNU/Linux distribution can sign a modified Release/InRelease file and if my machine somehow downloads that version of the release file, my machine could be made to download and install packages that the distribution didn t intend me to install. Further, it seems that anyone who has access to the main HTTP server, or any of its mirrors, or is anywhere on the network between them and my machine (when plaintext HTTP is used), can either stall security updates on my machine (on a per-IP basis), or use it to send my machine (again, on a per-IP basis to avoid detection) a modified Release/InRelease file if they had been able to obtain the private signing key for the archive. These are mighty powers that warrant overview.
I ve always put off learning about the processes to protect the apt infrastructure, mentally filing it under so many people rely on this infrastructure that enough people are likely to have invested time reviewing and improving these processes . Simultaneous, I ve always followed the more free-software friendly Debian-derived distributions such as gNewSense and have run it on some machines. I ve never put them into serious production use, because the trust issues with their apt package repositories has been a big question mark for me. The enough people part of my rationale for deferring this is not convincing. Even the simple question of is someone updating the apt repository is not easy to understand on a running gNewSense system. At some point in time the gNewSense cron job to pull in security updates from Debian must have stopped working, and I wouldn t have had any good mechanism to notice that. Most likely it happened without any public announcement. I ve recently switched to Trisquel on production machines, and these questions has come back to haunt me.
The situation is unsatisfying and I looked into what could be done to improve it. I could try to understand who are the key people involved in each project, and may even learn what hardware component is used, or what software is involved to update and sign apt repositories. Is the server running non-free software? Proprietary BIOS or NIC firmware? Are the GnuPG private keys on disk? Smartcard? TPM? YubiKey? HSM? Where is the server co-located, and who has access to it? I tried to do a bit of this, and discovered things like Trisquel having a DSA1024 key in its default apt trust store (although for fairness, it seems that apt by default does not trust such signatures). However, I m not certain understanding this more would scale to securing my machines against attacks on this infrastructure. Even people with the best intentions, and the state of the art hardware and software, will have problems.
To increase my trust in Trisquel I set out to understand how it worked. To make it easier to sort out what the interesting parts of the Trisquel archive to audit further were, I created debdistdiff to produce human readable text output comparing one apt archive with another apt archive. There is a GitLab CI/CD cron job that runs this every day, producing output comparing Trisquel vs Ubuntu and PureOS vs Debian. Working with these output files has made me learn more about how the process works, and I even stumbled upon something that is likely a bug where Trisquel aramo was imported from Ubuntu jammy while it contained a couple of package (e.g., gcc-8, python3.9) that were removed for the final Ubuntu jammy release.
After working on auditing the Trisquel archive manually that way, I realized that whatever I could tell from comparing Trisquel with Ubuntu, it would only be something based on a current snapshot of the archives. Tomorrow it may look completely different. What felt necessary was to audit the differences of the Trisquel archive continously. I was quite happy to have developed debdistdiff for one purpose (comparing two different archives like Trisquel and Ubuntu) and discovered that the tool could be used for another purpose (comparing the Trisquel archive at two different points in time). At this time I realized that I needed a log of all different apt archive metadata to be able to produce an audit log of the differences in time for the archive. I create manually curated git-repositories with the Release/InRelease and the Packages files for each architecture/component of the well-known distributions Trisquel, Ubuntu, Debian and PureOS. Eventually I wrote scripts to automate this, which are now published in the debdistget project.
At this point, one of the early question about per-IP substitution of Release files were lingering in my mind. However with the tooling I now had available, coming up with a way to resolve this was simple! Merely have apt compute a SHA256 checksum of the just downloaded InRelease file, and see if my git repository had the same file. At this point I started reading the Apt source code, and now I had more doubts about the security of my systems than I ever had before. Oh boy how the name Apt has never before felt more Apt?! Oh well, we must leave some exercises for the students. Eventually I realized I wanted to touch as little of apt code basis as possible, and noticed the SigVerify::CopyAndVerify function called ExecGPGV which called apt-key verify which called GnuPG s gpgv. By setting Apt::Key::gpgvcommand I could get apt-key verify to call another tool than gpgv. See where I m going? I thought wrapping this up would now be trivial but for some reason the hash checksum I computed locally never matched what was on my server. I gave up and started working on other things instead.
Today I came back to this idea, and started to debug exactly how the local files looked that I got from apt and how they differed from what I had in my git repositories, that came straight from the apt archives. Eventually I traced this back to SplitClearSignedFile which takes an InRelease file and splits it into two files, probably mimicking the (old?) way of distributing both Release and Release.gpg. So the clearsigned InRelease file is split into one cleartext file (similar to the Release file) and one OpenPGP signature file (similar to the Release.gpg file). But why didn t the cleartext variant of the InRelease file hash to the same value as the hash of the Release file? Sadly they differ by the final newline.
Having solved this technicality, wrapping the pieces up was easy, and I came up with a project apt-canary that provides a script apt-canary-gpgv that verify the local apt release files against something I call a apt canary witness file stored at a URL somewhere.
I m now running apt-canary on my Trisquel aramo laptop, a Trisquel nabia server, and Talos II ppc64el Debian machine. This means I have solved the per-IP substitution worries (or at least made them less likely to occur, having to send the same malicious release files to both GitLab and my system), and allow me to have an audit log of all release files that I actually use for installing and downloading packages.
What do you think? There are clearly a lot of work and improvements to be made. This is a proof-of-concept implementation of an idea, but instead of refining it until perfection and delaying feedback, I wanted to publish this to get others to think about the problems and various ways to resolve them.
Btw, I m going to be at FOSDEM 23 this weekend, helping to manage the Security Devroom. Catch me if you want to chat about this or other things. Happy Hacking!
Apart from it, the show also shows prejudice about caste and background. I wouldn t go much into it as it s worth seeing and experiencing.
The following contributors got their Debian Developer accounts in the last two months:
We are happy to share that the 
Hello folks!
We recently received a case letting us know that Dataproc 2.1.1 was unable to write to a BigQuery table with a column of type JSON. Although the BigQuery connector for Spark has had support for JSON columns since 0.28.0, the Dataproc images on the 2.1 line still cannot create tables with JSON columns or write to existing tables with JSON columns.
The customer has graciously granted permission to share the code we developed to allow this operation. So if you are interested in working with JSON column tables on Dataproc 2.1 please continue reading!
Use the following gcloud command to create your single-node dataproc cluster:
Nitrokey Start
(no site yet)
in October 2021. they generally seem to keep up with
shipping. update (august 2022): they rolled out a second line of
laptops (12th gen), first batch shipped, second batch 
),
and from time-to-time I ve made sure that my code runs on OpenBSD
too.
Platforms beyond that will likely not be supported at the expense of either
of those two. I ll take fixes for bugs that fix a problem on another
platform, but not damage the code to work around issues / lack of features
on other platforms (like Windows).
This is the second part of how I build a read-only root setup for my router. You might want to read 
